mlsanityData to 14 Oct 2026

When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

arXiv 2608.14610 ↗First posted 8 Jul 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 8 of its 23 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

8 of 23 references.

References · 8

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  4. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  5. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 170
  6. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 122
  7. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  8. 2023Lawyer LLaMA Technical Reportcited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
  2. 2026LexKairos: Benchmarking Legal Temporal Capabilities in LLMs
  3. 2025Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning
  4. 2026Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning
  5. 2026Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
  6. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  7. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
  8. 2025Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities