When two AI models get different personality-test scores, is the difference real? We checked 1,716 comparisons (a pair of models on one scale) across 13 models. Only 62 of them, 3.6%, survived two tests: the gap had to be larger than both models' own setup ranges, and the same model had to stay ahead in every matched run.
The two tests
- The difference between the two models in the main setup is larger than the larger of their two setup ranges, where the range is the highest minus the lowest score across ten temperature-0 setups.
- The two models keep the same strict order in every matched temperature-0 administration, with at least 8 matched.
Results
| step | comparisons |
|---|---|
| all model-pair-by-scale comparisons | 1,716 |
| gap larger than both setup ranges | 112 |
| same order in every matched run | 200 |
| both tests | 62 (3.6%) |
Of the 62, 26 are on Big Five scales, 13 on RIASEC, 1 on Dark Triad and 22 on Multiple Intelligences. One model, gemini-2.5-pro, is involved in 42 of the 62 and is the higher-scoring model in 26 of them. Without it, 20 comparisons pass both tests. The largest surviving difference is Multiple Intelligences bodily: qwen3-235b above gpt-oss-120b, 80 against 10 in the main setup, a gap of 3.14 human standard deviations, with the same order in all 10 matched administrations.
Habits separate models more than content does
- Agreeing with items whatever the wording (Big Five): at or above the 95th human percentile for gemma-3-27b, llama-4-maverick, deepseek-v3.2, mistral-large-3, qwen3-32b and kimi-k2.5; at or below the 5th for gpt-oss-120b and gpt-6-astra.
- Using the end points of the scale: at or above the 95th human percentile for gpt-oss-120b (70.0% of answers) and gemini-2.5-pro (96.0%).
- Answering every item of a test with the same option: deepseek-v3.2 did it on RIASEC and Multiple Intelligences in the main setup.
- Not answering in the requested form: under the reworded instruction, nova-pro returned something that was not an option on 67 of 168 single-item requests, so those administrations were set aside as invalid.
What the answers looked like
Across all 16 models, 91 of 96 main-setup RIASEC cells sit above the human median, so the models endorse more of the listed activities than the median test-taker, and all 16 score below the human median on Machiavellianism. Both readings sit on top of the habits above and on one setup, so they describe these answers and are not a trait.
What to do with this
Do not pick a model from a personality quiz screenshot. If a model comparison is reported, report the range across setups with it and say how many replies could not be read. To choose a model for a job, test it on that job.
What we are not saying
That the 62 differences are the only real ones. They are the ones that clear two fixed tests under one family of instructions. The count including the three subscription models is 206 of 2,640 (7.8%), and it is not used here because it mixes temperature-0 models with models that cannot be set to temperature 0. We say nothing about the five other instruments in the JobCannon dataset, which were not run.
FAQ
Which AI model is the most agreeable?
We do not rank models on a trait. On the Big Five, 26 of 1,716 comparisons passed both tests, and any ranking made from one setup would not be expected to hold under another.
Why do models differ so much on some scales?
Often because of how they answer, such as agreeing with everything or using only the end points, and not because of what the items say.
Is this a benchmark of which model is best?
No. It measures how stable personality-questionnaire scores are, and says nothing about quality on any task.
Data
huggingface.co/datasets/PeterKol/llm-questionnaire-protocol-sensitivity (commit 21b07690e79af51e489122501538541e75431c76, CC BY 4.0). The human comparison set is huggingface.co/datasets/PeterKol/jobcannon-psychometric-responses.