The Science Behind the Questions
Every time you answer a personality questionnaire item — "I enjoy being the center of attention" (Agree/Disagree) — something more sophisticated than it appears is happening. The item has been selected from thousands of candidates based on empirical evidence that it loads on a particular personality factor more than others. Its response scale has been designed to maximize variance and minimize ceiling effects. The pattern of your responses across dozens or hundreds of such items will be mathematically combined to produce trait scores. Those scores will be compared to normative data from thousands of previous respondents to situate your results in a population context.
This entire process is governed by a science called psychometrics — the measurement of psychological constructs. Understanding its basic principles helps you evaluate which personality tests deserve your trust and which do not.
Reliability: The Foundation of Measurement
Reliability refers to consistency of measurement. A personality test is reliable if it produces similar results when administered to the same person on different occasions (test-retest reliability), when different scorers score the same responses (inter-rater reliability), and when different items measuring the same trait produce consistent responses (internal consistency).
Reliability is expressed as a coefficient between 0 and 1. For personality assessments, reliability coefficients above 0.80 are generally considered acceptable for individual-level interpretation. The Big Five traits show test-retest reliabilities of 0.70-0.85 over periods of weeks to months.
Without adequate reliability, all subsequent claims about what a test measures are meaningless. You cannot validly measure something you cannot measure consistently.
Validity: Does It Measure What It Claims?
Validity is more complex than reliability. A test can be highly reliable (consistent) while measuring the wrong thing. Validity asks: does this test actually measure the psychological construct it claims to measure, and does it predict the outcomes it is supposed to predict?
Content validity: Do the items adequately cover the domain being measured? A Big Five Extraversion scale with only items about talkativeness would have poor content validity, because Extraversion also encompasses activity level, positive emotionality, and social dominance.
Construct validity: Does the test behave as predicted by the theory of the construct? If Extraversion exists and the test measures it, extrovert scores should predict more social activity, faster friendship formation, and preference for social environments — and they do.
Predictive validity: Does the test score predict relevant real-world outcomes? Big Five Conscientiousness scores predict job performance; this is predictive validity. RIASEC scores predict occupational choice; this is also predictive validity.
Factor Analysis: How Personality Dimensions Are Discovered
How did researchers identify the Big Five dimensions in the first place? Through a statistical method called factor analysis, which identifies groups of items that correlate strongly with each other (suggesting they measure a common underlying construct) and minimally with other groups.
Starting with thousands of personality-descriptive words, researchers find that these descriptors cluster into a small number of groups. Items like "talkative," "assertive," "active," and "sociable" cluster together and define Extraversion. Items like "organized," "disciplined," "reliable," and "thorough" cluster together and define Conscientiousness. This empirical approach — letting data reveal structure rather than imposing it theoretically — is why the Big Five is more robust than theoretically derived systems like Myers-Briggs.
Norms: Making Scores Meaningful
A raw score on a personality scale (say, 34 out of 50 items endorsed) is meaningless without normative context. Normed scores tell you where you fall relative to a reference population. Percentile scores (you are at the 72nd percentile on Extraversion) are most interpretable — 72% of the normative sample scored lower than you.
Good assessments publish their normative data, specify the population on which norms are based, and update norms periodically. Norms based on a non-representative sample (only college students, for example) may not accurately describe the broader population.
Evaluating Personality Tests
Questions to ask when evaluating a personality test:
- Is there published research on this test's reliability and validity?
- Are the reliability coefficients above 0.75?
- Does the test predict outcomes it claims to predict (predictive validity)?
- Are normative data available, and are they from a relevant population?
- Has the test been validated across cultures and demographic groups?
The Big Five assessments at JobCannon use validated IPIP item pools with published psychometric properties. Take the Big Five test to see for yourself.