# Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

arXiv: https://arxiv.org/abs/2605.23497 · HTML: https://mlsanity.com/paper/2605.23497/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 22 May 2026.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 9 of 9)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [A Reasoning-Focused Legal Retrieval Benchmark](https://mlsanity.com/paper/2505.03970/) (2025)
- [Is Your LLM Outdated? A Deep Look at Temporal Generalization](https://mlsanity.com/paper/2405.08460/) (2024)
- [Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking](https://mlsanity.com/paper/2509.11353/) (2025)
- [LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation](https://mlsanity.com/paper/2510.07243/) (2025)
- [Do LLMs Truly Understand When a Precedent Is Overruled?](https://mlsanity.com/paper/2510.20941/) (2025)
- [Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models](https://mlsanity.com/paper/2509.04304/) (2025)

## Built on by (most cited first, 1 of 1)

- [Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law](https://mlsanity.com/paper/2608.09393/) (2026)

## Abstract (arXiv)

Large language models are increasingly used for legal research, yet their fixed training cutoffs and reliance on static parametric knowledge are at odds with the evolving nature of statutory law. We study two temporal failure modes: post-cutoff staleness, where models apply superseded rules after legislative amendments, and recency bias, where models prefer newer provisions even when a historical version governs the fact pattern. To this end, we present a benchmark of 312 expert-validated, time-sensitive German statutory QA pairs spanning three categories: Post-Cutoff Amendment Questions, Pre-Amendment Questions, and Multi-Provision Pre-Amendment Questions. We evaluate five LLMs by OpenAI, Anthropic and DeepSeek under four inference settings: Vanilla, Web-search, and two retrieval-augmented variants that enforce temporal validity via a fact date extraction and version filtering. Using an LLM-as-a-judge validated against human expert ratings, we find severe degradation in the Vanilla post-cutoff setting. Both RAG approaches substantially improve performance across all question types, while web search yields unstable gains and exhibits a marked recency bias on historically anchored tasks. Our results indicate that reliable legal QA requires treating temporal validity as a hard constraint.