mlsanityData to 7 Oct 2026

The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance

arXiv 2406.11634 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Cloze testing is a common method for measuring the behavior of large language models on a number of benchmark tasks. Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We find that counterfactual prompting does sufficiently mitigate the BRP effect. The BRP effect is found to have a similar effect to test taking strategies employed by humans leading to the conflation of task performance and test-taking ability. We propose the Nvr-X-MMLU task, a variation of MMLU, which helps to disambiguate test-taking ability from task performance and reports the latter.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

17 of 22 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  6. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  7. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  8. 2024Are We Done with MMLU?cited by 178
  9. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  10. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  11. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  12. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
5 more
  1. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  2. 2021Do language models learn typicality judgments from text?cited by 32
  3. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  4. 2023Using Artificial Populations to Study Psychological Phenomena in Neural Modelscited by 8
  5. 2024Do Large Language Models Learn Human-Like Strategic Preferences?cited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slowcited by 8
  2. 2024Large Language Model Recall Uncertainty is Modulated by the Fan Effectcited by 7
  3. 2025LLMs as Agentic Cooperative Players in Multiplayer UNOcited by 0