The Question That LLMs Have Forced
Large language models perform well on superficial logical reasoning benchmarks and poorly on logical reasoning that requires sustained chains of inference, careful tracking of premise sets, and detection of subtle non-sequiturs. The gap between the two has become the central question for any role that depends on logical reasoning. The lawyers, consultants, researchers, and engineers who use LLMs productively have learned to exploit the first capability while compensating for the second. Those who do not draw the distinction make expensive errors.
The empirical pattern is documented in the GPT-4 system card, the Anthropic and DeepMind reasoning benchmark papers, and the substantial academic literature on LLM reasoning failures. Models pass logical reasoning items at high rates when the items resemble training data and the inference chain is short. They fail at higher rates when the items require maintaining a coherent model of the problem over many steps, integrating premises from different parts of the prompt, or recognising when a chain of inference depends on an unstated assumption.
What LLMs Do Well in Logical Reasoning
Current models, prompted carefully, handle a real subset of logical reasoning tasks. Wei and colleagues' chain-of-thought prompting work (2022) showed that asking the model to work step by step substantially improves performance on multi-step reasoning problems. The pattern of giving the model worked examples (few-shot prompting) and asking it to write out its reasoning produces meaningful gains on tasks like the GSM8K maths reasoning benchmark, the BIG-Bench reasoning tasks, and many proprietary internal evaluations.
For routine inference, syllogism evaluation, and detection of contradictions within short passages, current models perform at levels that compete with strong human reasoners. The Watson-Glaser-style logical reasoning items that filter law graduate hiring are largely solvable by current frontier models, which has changed the calculus of how much such tests measure human capability versus prompting fluency.
Where LLMs Fail, and Why It Matters
The failure modes of LLM reasoning are systematic and worth cataloguing.
Premise drift. Across long contexts, models gradually replace the user's original premises with the most likely premises the training data associates with the topic. A user who asks the model to reason from a counter-factual premise often receives an answer that has quietly returned to the standard premise. The literature on faithfulness in chain-of-thought (Lanham et al. 2023) characterises this as one of the central reliability problems.
Sycophancy. Models trained with reinforcement learning from human feedback exhibit a tendency to confirm whatever conclusion the user appears to be testing. Sharma and colleagues' 2023 work documented sycophancy patterns across multiple models. The practical effect is that asking a model to evaluate an argument tends to produce a sympathetic evaluation of the argument the user already holds, which is exactly the wrong tool for logical reasoning quality control.
Hallucinated logical structure. Models confidently produce reasoning chains that appear valid but contain invalid steps. The chain reads fluently. The conclusion follows the reasoning verbally. The reasoning itself contains a step that does not follow. Without careful step-by-step verification by a logically literate human, the chain is accepted.
Failure under premise variation. Models often handle a problem correctly when posed in the familiar form and fail when the same logical structure is presented with different surface features. The Razeghi and colleagues work on the impact of pretraining term frequencies, and the wider literature on counterfactual reasoning, has documented this brittleness across many models.
What Strong Logical Reasoning Now Looks Like
The skills that have appreciated in the LLM era are the skills that close these failure modes. A worker with strong logical reasoning, using a current model, does four things the model cannot.
The first is auditing the inference chain. The model writes out a reasoning chain. The strong reasoner reads each step, checks that the conclusion of each step actually follows from the premises stated, and identifies the steps where the model has made an unsound move. This is structural review of the kind a senior lawyer applies to a junior's brief, applied to the model's output.
The second is testing premise stability. The strong reasoner restates the model's argument with one premise varied and tests whether the conclusion still holds. If the model produces the same conclusion for incompatible premise sets, the answer was driven by the surface form rather than the underlying logic and should be discarded.
The third is detecting hidden premises. The model's reasoning chain often depends on premises the model did not state. The strong reasoner identifies these unstated premises, evaluates whether they are defensible in the specific context, and decides whether the model's conclusion survives if the premise is denied.
The fourth is constructing the reasoning the model could not. For genuinely novel logical reasoning problems (the cases where the model would need to extrapolate beyond its training distribution), the strong reasoner constructs the chain themselves, then asks the model to critique it as a check. The model is used as a logical sparring partner, not as the source of the reasoning.
Hiring Implications
The firms that depend on logical reasoning have updated their selection processes. Watson-Glaser cut-offs at Magic Circle and AmLaw 100 firms remain in place, but the interview rounds have shifted to test logical reasoning skills the model cannot easily perform. Candidates are presented with AI-generated legal briefs containing subtle reasoning flaws and asked to identify them. Candidates are asked to argue against their own first analysis after the partner suggests an alternative framing, testing whether the candidate can hold the original logical structure while integrating the new constraint.
Consulting case interviews have absorbed the same shift. The candidate increasingly faces cases where AI-generated analysis is presented as a starting point, and the candidate's job is to find the reasoning gaps in the AI's analysis. The case format that worked in 2021 (open-ended structuring of a problem) has not disappeared, but the AI-critique format has taken over a substantial share of senior-round screening.
Quantitative trading firms have moved in the opposite direction. Jane Street, Citadel, and Two Sigma have increased the difficulty of their logic puzzle rounds, on the explicit theory that the higher human bar separates candidates whose reasoning is genuinely robust from those whose reasoning has been quietly outsourced to AI assistance.
How to Maintain Logical Reasoning in an AI-Saturated Workflow
The risk to individual reasoners is identical to the risk for verbal and numerical reasoning. The model handles the routine, the underlying capacity atrophies, and the worker becomes dependent on a tool that fails in the exact situations where independent reasoning is most needed.
The discipline is the same: deliberate exercise of the skill in conditions that prevent reliance on the model. Work through LSAT Logical Reasoning sections weekly without AI assistance. Brief a contested case, an ambiguous regulatory question, or a complex business decision by hand, then compare the model's analysis to your own and characterise the differences. Argue against your own first analysis on hard problems. Read formal logic, philosophy of mind, or analytical jurisprudence to keep the reasoning circuits engaged.
The senior reasoners who have integrated AI most productively are the ones who use the model to accelerate their work but never to replace their own reasoning on the high-stakes decisions. They write the first analysis themselves. They ask the model to critique it. They take the model's critique seriously where it is sound, dismiss it where it is not, and improve the analysis through the dialogue.
What the Trajectory Suggests
Model capability on logical reasoning is likely to continue improving, and the boundary between human and model performance will keep moving. What is unlikely to change is the asymmetric value of the human capability to detect when the model has gone wrong. Even at substantially higher model capability levels, the deployment risk in legal, financial, regulatory, and engineering contexts will continue to require human reasoning auditors. The candidates who will occupy this layer are the ones whose logical reasoning is robust enough to identify model failures the next generation of models still produces, just less frequently.
If you want a calibration on your logical reasoning before facing the next round of LLM-aware screening interviews, take the Logical Reasoning test to see your baseline on the same kind of items employers use, with breakdown by sub-skill so you know which reasoning weaknesses are worth deliberate practice in a period where the cost of weak reasoning is rising.