# The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

arXiv: https://arxiv.org/abs/2606.17609 · HTML: https://mlsanity.com/paper/2606.17609/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning](https://mlsanity.com/paper/2408.14398/) (2024)
- [Self-Correcting Large Language Models: Generation vs. Multiple Choice](https://mlsanity.com/paper/2511.09381/) (2025)
- [M-Wanda: Improving One-Shot Pruning for Multilingual LLMs](https://mlsanity.com/paper/2505.21171/) (2025)

## Built on by (most cited first, 2 of 2)

- [ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation](https://mlsanity.com/paper/2607.13124/) (2026)
- [SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models](https://mlsanity.com/paper/2609.34044/) (2026)

## Abstract (arXiv)

Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.