# LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories

arXiv: https://arxiv.org/abs/2509.20909 · HTML: https://mlsanity.com/paper/2509.20909/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Rethinking LLM Memorization through the Lens of Adversarial Compression](https://mlsanity.com/paper/2404.15146/) (2024)
- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [DE-COP: Detecting Copyrighted Content in Language Models Training Data](https://mlsanity.com/paper/2402.09910/) (2024)
- [Language Model Inversion](https://mlsanity.com/paper/2311.13647/) (2023)
- [Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method](https://mlsanity.com/paper/2409.14781/) (2024)
- [Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation](https://mlsanity.com/paper/2309.10677/) (2023)
- [Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models](https://mlsanity.com/paper/2311.06233/) (2023)
- [Data Contamination Can Cross Language Barriers](https://mlsanity.com/paper/2406.13236/) (2024)
- [Data Contamination Calibration for Black-box LLMs](https://mlsanity.com/paper/2405.11930/) (2024)
- [Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation](https://mlsanity.com/paper/2406.13990/) (2024)
- [CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models](https://mlsanity.com/paper/2311.09154/) (2023)
- [Do LLMs Know to Respect Copyright Notice?](https://mlsanity.com/paper/2411.01136/) (2024)
- [Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation](https://mlsanity.com/paper/2501.18771/) (2025)
- [Memorization vs. Reasoning: Updating LLMs with New Knowledge](https://mlsanity.com/paper/2504.12523/) (2025)
- [DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning](https://mlsanity.com/paper/2406.04197/) (2024)
- [Measuring Copyright Risks of Large Language Model via Partial Information Probing](https://mlsanity.com/paper/2409.13831/) (2024)
- [Memorization or Reasoning? Exploring the Idiom Understanding of LLMs](https://mlsanity.com/paper/2505.16216/) (2025)

## Built on by (most cited first, 2 of 2)

- [TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance](https://mlsanity.com/paper/2608.08139/) (2026)
- [HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix](https://mlsanity.com/paper/2610.01170/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are commonly evaluated on challenging benchmarks such as AIME and Math500, where benchmark contamination can make memorized solutions appear as genuine reasoning. Existing detection methods largely rely on surface overlap, completion behavior, or final-output likelihood, and often degrade when inputs are simply rephrased. In this paper, we propose LogitTrace(Layerwise Logit Trajectories), a framework for analyzing memorization-like decision dynamics through intermediate logit trajectories. Instead of judging memorization only from the final answer, LogitTrace examines how model preferences emerge and stabilize across layers. We find that contaminated examples tend to show earlier commitment, while clean examples exhibit more gradual evidence accumulation. These trajectory signals allow a lightweight classifier to separate contaminated and clean examples across multiple models and input variants. Controlled LoRA injection experiments further show that repeated exposure to target samples induces similar trajectory patterns. Overall, our results suggest that LogitTrace provides evidence beyond surface overlap and final-output confidence, offering a useful lens for studying memorization-like behavior in LLMs.