Human baseline data for AI and LLM evaluation
A benchmark score means one thing on a leaderboard and another next to how people actually answer the same item. JobCannon holds 671,760 completed psychometric and cognitive assessments, and we place model output inside that human distribution instead of asking another model to judge it. Below: how the comparison works, what you get, and how a lab starts.
Why a model score needs a human distribution behind it
A model answering a reasoning item, a personality item or a language item produces a number. On its own, that number says nothing: is it high, low, typical, or nowhere near how a person responds? The only way to read it is against how real people score on the same items under the same scoring rules.
That is a different question from LLM-as-a-judge, where one model grades another model’s free-text answer. A human baseline compares a model with people who took the test. LLM-as-a-judge compares it with a second model’s opinion. Neither replaces the other.
What the baseline is built from
Snapshot dated 2026-09-30, the same one /research/data publishes.
- 202 published instruments, 154 of them with stored results: cognitive tests, personality inventories and interest inventories, each with its own scoring rules.
- Responses in 31 interface languages, from 220 countries and territories.
- Completion time measured, not estimated: median 3 min 25 s, middle half between 1 min 53 s and 6 min 12 s, over 634,815 timed results.
- Cronbach’s alpha published per scale for 20 assessments, weak scales included.
- 23 of the 202 instruments implement a published, peer-reviewed model.
The live counter on /research/data is always higher. Cite the dated snapshot: it still reconciles in a paper months later.
How the comparison works
We score a model’s answers on the same items and the same scale a person faces: a cognitive item with one correct answer, or a personality item with none. The model’s score is then placed inside the human distribution for that instrument, computed with the rules our own respondents are scored by. The output is where the model falls relative to people, not a verdict on the model.
This is not a proposal on paper. We have already run it on a panel of 13 widely used AI models against our cognitive item bank, with repeated runs and two checks for models that follow answer letters or remember the wording: one re-orders the answer options, the other rewrites the question text. What it found is in the next section.
We also published a separate open-data study of how test setup moves AI personality-test scores: the test-setup sensitivity study.
What the comparison found
Do people and AI models find the same questions hard? We checked on 395 questions from our reasoning test. For each question we counted how many people answered it correctly and how many of 13 AI models did. Each question was answered by 728 to 866 people, 15,959 English-language sessions in total.
The main points
- People answer 74% of the questions correctly on average, the models 92%.
- Questions that are hard for people are only partly hard for models. The match is 0.30 on a scale where 1 means “the same questions” and 0 means “no relation”.
- How demanding a question is predicts how hard it is for people about twice as well as for models.
- The result holds with people in all languages (120,528 sessions) and with the models run three times.
What makes a question hard
- People are held back by maths, logic, reading comprehension and long question text. Models barely are.
- Both people and models are tripped up by questions that need you to know what you don’t know, and to separate what matters from what doesn’t.
- Unusual, unfamiliar questions are harder for models than for people.
Split into thirds by how hard the questions are for people
| Questions | People correct | Models correct |
|---|---|---|
| Hardest third for people | 53% | 87% |
| Middle third | 78% | 92% |
| Easiest third for people | 90% | 97% |
Limits
- The people are volunteers on our site, unsupervised. They are not a random sample of the population.
- How demanding each question is was annotated by an AI model, not by people. On questions experts had already annotated, it mostly agreed with them.
- The questions are public, so models may have seen them in training. We checked by shuffling the answer options and rewriting the questions; the models’ results did not change.
- The strongest models get almost everything right, so this test cannot tell the best of them apart well.
- We do not publish data on individual questions.
Exact figures
Spearman correlations over 395 items, 95% percentile bootstrap intervals over items (10,000 draws). Mean annotated demand vs share of people correct: −0.49 [−0.57, −0.40]; vs share of models correct: −0.26 [−0.35, −0.17]. People vs models: 0.30 [0.21, 0.39]. People in all interface languages: −0.51, −0.26, 0.33. Majority of three samples at temperature 1.0 (12 models): −0.49, −0.25, 0.27.
Demand is the mean level (0 to 5) across the 18 demand scales of ADeLe, annotated by gemini-2.5-flash. On items from the published ADeLe annotation the judge reproduced the authors’ levels at quadratic-weighted kappa 0.75 (n = 955). Scales that go with difficulty for both: calibrating knowns and unknowns (people −0.39, models −0.42), identifying relevant information (people −0.30, models −0.31). Formal-science knowledge, quantitative and logical reasoning, verbal comprehension and volume: people −0.17 to −0.29, models −0.12 to 0.06. Atypicality: people −0.17, models −0.32.
People: each person’s first English-language session, at least 60 seconds long, with answers that rebuild exactly to the published score. Models: one run per item at temperature 0, graded with the key that scores people. 5 of the 400 items are excluded: 1 without a demand annotation, 4 whose answer key was corrected in September 2026. Probes with re-ordered options and rewritten questions on 12 of the 13 models: no accuracy change survives Holm correction in any model. All 13 models answered 266 of the 395 items correctly, which caps any correlation with model accuracy. Descriptive only: no factor model was fitted. Analysis dated 2026-09-30.
How a lab works with us
One email: which model or models you want evaluated, on which instruments and in which languages. We run them through the same items and send a comparison report: each model’s scores and profile placed against the human distribution, with a written analysis. The report is computed output. It does not include raw responses, per-row data or item-level statistics.
Write to partners@jobcannon.io. We answer by email.
What we deliver
The aggregates published on this site are licensed CC BY 4.0. What a lab receives from us is a computed evaluation report for its own model: score, profile, percentile placement against the human distribution, and a written analysis. Raw responses, per-row data and item-level statistics are never part of it.
Methodology and sample
The instruments are documented on /methodology, per-scale reliability is on /research/reliability, and the Big Five has its own technical manual. The sample is naturalistic: self-selected, unproctored, unpaid and not quota-balanced by age, sex, education or country. That is what makes it a strong human baseline for measurement comparison: real people answering for their own reasons, in 31 languages.
Common questions
What is a human baseline for AI evaluation?
The distribution of scores real people get on the same items a model is evaluated on, scored the same way. A model's number only means something once it sits inside that distribution: typical, unusual, or nowhere near how people respond.
Why not just use LLM-as-a-judge?
It measures agreement between models, not how people behave. A model can rate another model's answer highly without either resembling a real response pattern. A human baseline anchors the comparison to actual respondents.
How large is the baseline, and is it only English?
As of 2026-09-30: 671,760 completed assessments across 154 instruments, in 31 interface languages, from 220 countries and territories. The live figure is higher; the dated snapshot is the citable one.
How do I get access?
Email partners@jobcannon.io with which model or models you want evaluated, on which instruments and in which languages. We reply by email with what is possible.
Tell us what you want to evaluate
One email with what you want to evaluate, in which languages, at roughly what scale. We reply by email.