All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting
Not on the map. Why?
Abstract (arXiv)
Backtesting LLMs on resolved events assumes models reason only from pre-cutoff knowledge, yet pretrained models inevitably leak post-cutoff knowledge. We introduce a claim-level evaluation framework that decomposes prediction rationales into atomic claims and applies Shapley values to quantify each claim's decision impact, yielding Shapley-DCLR (Shapley-weighted Decision-Critical Leakage Rate) -- an interpretable metric measuring what fraction of decision-driving reasoning is contaminated. We further propose TimeSPEC (Time-Supervised Prediction with Extracted Claims), an inference-time architecture that interleaves temporally-filtered retrieval with claim-level supervision, producing predictions grounded entirely in pre-cutoff evidence. Across three LLMs, the ablation experiments confirm retrieval and supervision are jointly necessary; and a three-task probe further illstrates that the performance cost of temporal enforcement scales with each task's reliance on post-cutoff information.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 19 Feb 2026.
Builds on
18 of 32 references.
References · 18
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023BloombergGPT: A Large Language Model for Financecited by 923
- 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
- 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
- 2018The Fact Extraction and VERification (FEVER) Shared Taskcited by 108
- 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
6 more
- 2022Forecasting Future World Events with Neural Networkscited by 63
- 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
- 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
- 2024Provably Accurate Shapley Value Estimation via Leverage Score Samplingcited by 23
- 2025Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agentscited by 13
- 2025LLMLagBench: Identifying Temporal Training Boundaries in Large Language Modelscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.