Why I Built WorldCal
I love AI.
That is probably the most important thing to say at the beginning.
I think these models are fascinating. I think they have the potential to be genuinely revolutionary, and I want them to become extraordinarily good at understanding the world and helping the people who live in it.
WorldCal came out of wanting to push on one very small piece of that problem.
The question behind the first study was simple: when an AI model invents an ordinary couple, what kind of couple does it imagine?
And, more importantly, can we measure the answer against something real?
That last part matters to me.
There are enormous philosophical questions around representation. Should an AI model reproduce society exactly as it exists? Should it deliberately make the world it generates more inclusive? Should historically underrepresented groups appear more frequently than their population share? What does “fair” representation even mean?
Those are important questions. I also think they are incredibly hard.
For this first project, I wanted to avoid pretending that I knew the answers.
Instead, I wanted to treat part of the problem as an engineering problem.
Start with the mirror
One way to think about an AI model is as a mirror.
If I ask a model to represent the United States, how accurately can it represent the people who actually live here?
That is different from asking what the model should represent.
Maybe we eventually want a rose-colored mirror. Maybe there are good reasons to deliberately produce a world that is more inclusive, more balanced, or more aspirational than reality. I think that is a legitimate conversation.
But before getting there, I want to know whether the mirror works.
If we have high-quality demographic data describing a population, and we ask an AI system to represent that population, can it do it?
That is a much cleaner question.
For this study, I used American Community Survey data to anchor the model outputs to actual married and unmarried-cohabiting couple households in the United States. The Census Bureau has already done the extraordinarily difficult work of collecting that data. I do not need to decide what the “correct” moral representation of a couple should be. I can act as an intermediary between the models and the available data and ask how closely the two line up.
I like that framing because it is measurable.
And I think accurate representation of reality is a useful capability even if we later decide that some applications should intentionally depart from it.
Why this matters to me
I am also a gay man.
So yes, there is a personal reason I care about whether these systems represent same-sex couples accurately.
This project also grew out of a long-running set of conversations with Ryan. We had been talking for about a year about putting together an AI talk series for a StartOut, and those conversations kept circling around the same basic question: if AI is going to become this influential, what does that mean for the gay community? Is it actually representing us well?
We had looked at research on bias in areas like hiring, HR systems, healthcare, and other high-impact uses of AI. I kept trying to think through if there was a tiny spike I could do to investigate a sliver of the problem.
That back-and-forth helped push me toward this study. Once I ran the first exploratory analyses, the results were surprising enough that I felt like I had to keep going. From there, the research started driving itself: how do I make the test cleaner, how do I tie the model outputs back to something real, and how do I turn an interesting result into something that can actually be measured and repeated?
So a real shout-out to Ryan for the conversations, the questions, and the early push in a productive direction.
AI models are increasingly generating examples, stories, recommendations, simulated people, synthetic data, educational material, marketing copy, and a huge amount of the text people interact with every day.
If whole categories of people quietly disappear from the model's default version of the world, that matters.
At the same time, I don't want WorldCal to become a vehicle for making political declarations about individual models. I would much rather measure the behavior carefully, publish the records, explain exactly how I arrived at the result, and let people decide what they think it means.
My contribution can be much simpler:
Here is what the model generated. Here is what the real-world data says. Here is how far apart they are.
That feels useful.
The prompts became one of the most interesting parts
One of the things I did not fully appreciate when I started this work was how much the structure of the prompt itself could matter.
During exploratory work, I had a set of prompts involving couples raising children. The behavior looked very different from the ordinary adult-couple prompts.
I ended up quarantining those prompts and excluding them from the confirmatory study.
At the time, it felt mostly like a methodological decision: the parenting prompts were behaving differently and I did not have an exactly matched demographic denominator for them.
Looking back, I think that decision exposed something more interesting.
The language of the prompt can change the representational regime of the model.
We saw an even clearer example in the final results. For Claude Opus 5, tiny differences between two paraphrases of ostensibly the same scenario produced dramatically different same-sex representation rates.
That is fascinating.
It means representation is probably not some simple internal dial where a model has a fixed probability of generating a same-sex couple. The exact wording, context, relationship structure, and implied social situation can all push the model in different directions.
That also made me think differently about prior research.
Daniel Wang and his coauthors studied demographic deviation by explicitly asking models to assign attributes such as sexual orientation. Their results can look almost opposite to what I observed: once sexual orientation is explicitly raised, models can produce very large percentages of minority orientations.
I think there may be an intuitive explanation worth testing.
If I ask you to tell me someone's sexual orientation, I have already made sexual orientation salient.
In ordinary life, we often don't explicitly announce that a person is heterosexual. Calling attention to sexual orientation may itself suggest that there is something non-default to describe.
I don't know whether that is actually what is happening inside these models. It could come from training data, post-training, instruction following, or something else entirely.
But it gives us a testable hypothesis:
Does representation change depending on whether identity is unmarked, contextually salient, or explicitly requested?
That feels like a very interesting direction for the next round of research.
The calibration experiment made this even stranger
One of my favorite parts of the study is the calibration arm.
The main prompts basically say: write a realistic story about this couple.
The calibration prompts instead tell the model to imagine a couple household selected randomly from the relevant U.S. population.
Those sound similar.
The models do not always treat them similarly.
Claude Opus 5 generated same-sex married couples at 4.8% in the ordinary narrative condition, compared with an ACS reference of about 1.35%. When explicitly asked to imagine a randomly sampled married household, that rate fell to 0.5%.
Llama 4 Maverick went the other direction for unmarried couples: from 2.4% in ordinary narratives to 5.5% in the random-household condition, almost exactly on top of the ACS reference of about 5.74%.
That is wild to me.
It suggests that “tell me a story about the world” and “represent the statistical world” may be different capabilities inside the same model.
I want to understand that much better.
This should become a dashboard
The obvious next step is not another static paper.
I want this to become a measurement system.
Keep a stable set of prompts. Keep the methodology public. Keep the demographic references explicit. Run new models through the same process as they are released.
Then watch what happens over time.
Do models become more calibrated?
Do they become less sensitive to tiny wording changes?
Does a new model generation suddenly move dramatically closer to reality?
Do models that perform well on one population fail badly on another?
The first study focuses on same-sex couples because that is where I started, but the underlying methodology could be much broader.
There are demographic distributions for gender, geography, age, household structure, political behavior, and many other characteristics. Some will be much harder to measure cleanly than others, and every new area will need its own careful methodology.
But I love the general idea:
take model-generated worlds and connect them back to measurements of the actual world.
If we can do that consistently, we can start measuring something that otherwise stays fuzzy.
The human part is still important
Another thing this project gave me a new appreciation for is annotation.
Shout out to everyone who annotates datasets for a living.
It is hard.
I reviewed a lot of generated records myself. Hervé also spent a good amount of time independently reviewing records so that I could sanity-check the automated judging system.
That human review gave me much more confidence in the result.
LLMs as judges are incredibly useful. They make a study at this scale possible for one person working nights and weekends.
But I don't think the answer is to remove humans from the loop.
I think the interesting future is probably a strong interplay between both: models doing the enormous first pass, with humans carefully validating the areas where the measurement matters most.
I would love to have more independent annotators in future runs.
That takes time and money.
Which brings me to another constraint.
Independent research is expensive
This project is self-funded.
I also have a day job.
Every additional model call costs money. More prompts cost money. More generations cost money. Better human validation costs money. And all of this takes time.
I would love to simply run ten times as many experiments.
I can't.
At least not yet.
So part of the next phase of WorldCal is figuring out how to make this sustainable. Maybe that means funding. Maybe collaboration. Maybe more efficient experiment design. Probably some combination of all three.
For now, I want to keep building it because I think the questions are worth asking.
The first pancake
I also wanted to get this out quickly.
Publishing anything is hard. It is incredibly easy to keep refining, keep adding another check, keep rewriting one more section, and never actually put the work into the world.
I think of this as the first pancake.
The first pancake is never perfect. You make it, you see what went wrong, you adjust the heat, and the next one is better.
That is how I want to approach WorldCal. This study is small. There are things I will improve. I will find better prompts, better validation approaches, better ways to present the data, and probably things I did not even realize I should have been measuring yet.
But getting the first one out matters.
I would rather publish something careful and useful, learn from it in public, and refine the process over time than spend forever trying to make the first attempt perfect. The goal is for this to be the beginning of a measurement program, not the final word on anything.
We'll see how the next pancakes turn out.
I want AI to get better
There is an irony I really enjoy about using AI to study AI.
ChatGPT and Cursor were enormously helpful throughout this project. I used them for coding, debugging, thinking through methodology, analyzing results, challenging assumptions, and editing the final paper.
I love these tools.
I'm not doing this research because I think AI is bad.
I'm doing it because I think AI can be much better.
I want models that understand reality extraordinarily well. I want them to understand people accurately. I want them to handle complexity rather than collapsing the world into a small collection of defaults.
And I want researchers and engineers to have ways to tell when that is happening.
WorldCal is a very small attempt to build one of those measurements.
The first study is done.
Now I want to keep running it, keep adding to it, and see where the models go.
If this work can push them even a tiny bit toward being better representations of the world—and ultimately better tools for the humans who use them—I'll be very happy I spent the nights and weekends on it.