Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
Not on the map. Why?
Abstract (arXiv)
We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B". The failure reproduces at first observation, but its empirical surface is unstable: between March and April 2026 several failure cells closed silently under the same model alias, with no version bump (GPT-5.4 on construction insurance moved from 96.6% to 100%, same prompt and harness). For regulated workflows, frontier-model accuracy is a moving compliance boundary that shifts without notice. We present the Aethis Eligibility Module, a neuro-symbolic architecture in which LLMs author rules from authoritative sources and an SMT-based layer executes them deterministically, consistent with the authored specification regardless of model drift, reasoning-effort defaults, or prompt format. Three evidence bases: (i) a controlled benchmark of 225 scenarios across four regulatory domains documents the pattern and, in replication, the drift that partially closed it; (ii) a 20-scenario adversarial extension on construction insurance, where the engine scores 20/20, as does one of four frontier configurations (GPT-5.4 at low reasoning effort), while the other three, including Anthropic's strongest model at evaluation time, fail the same coverage-gap edge case; (iii) external validation on nine peer-reviewed LegalBench tasks, 949 held-out cases, where the engine is significantly more accurate than all three frontier models (combined McNemar's p <= 0.003), with margins up to +41 points on the curated multi-prong tasks against the Anthropic models. The contribution is to relocate uncertainty from the inference boundary, where it is silent, to the specification boundary, where it is deliberate and audited. All scenarios, rule encodings, and results are public and reproducible.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 27 references that are in the map. First posted on arXiv 25 Jul 2026.
Builds on
12 of 27 references.
References · 12
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
- 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
- 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
- 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 168
- 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 94
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
- 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications
- 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
- 2026How Robustly do LLMs Understand Execution Semantics?
- 2026Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning
- 2025MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
- 2026When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
- 2026The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability