Resurrecting saturated LLM benchmarks with adversarial encoding
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Recent work showed that small changes in benchmark questions can reduce LLMs' reasoning and recall. We explore two such changes: pairing questions and adding more answer options, on three benchmarks: WMDP-bio, GPQA, and MMLU variants. We find that for more capable models, these predictably reduce performance, essentially heightening the performance ceiling of a benchmark and unsaturating it again. We suggest this approach can resurrect old benchmarks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. First posted on arXiv 10 Feb 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMs
- 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
- 2025LastingBench: Defend Benchmarks Against Knowledge Leakage
- 2025VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI
- 2026Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
- 2024MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
- 2025Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- 2026SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?