mlsanityData to 14 Oct 2026

Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods

arXiv 2403.00998 ↗First posted 1 Mar 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

This paper systematically compares different methods of deriving item-level predictions of language models for multiple-choice tasks. It compares scoring methods for answer options based on free generation of responses, various probability-based scores, a Likert-scale style rating method, and embedding similarity. In a case study on pragmatic language interpretation, we find that LLM predictions are not robust under variation of method choice, both within a single LLM and across different LLMs. As this variability entails pronounced researcher degrees of freedom in reporting results, knowledge of the variability is crucial to secure robustness of results and research integrity.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

29 of 51 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
17 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  2. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  3. 2022Using cognitive psychology to understand GPT-3cited by 398
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  5. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  6. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  7. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  8. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  9. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  10. 2021Language Model Evaluation Beyond Perplexitycited by 78
  11. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 70
  12. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  13. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
  14. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  15. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  16. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 23
  17. 2023Overinformative Question Answering by Humans and Machinescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  3. 2024Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Thinkcited by 14
  4. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  5. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processcited by 8
  6. 2024Bayesian Statistical Modeling with Predictors from LLMscited by 7
  7. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
  8. 2024Compare without Despair: Reliable Preference Evaluation with Generation Separabilitycited by 4
  9. 2025Can AI Make Energy Retrofit Decisions? An Evaluation of Large Language Modelscited by 1
  10. 2026Instructions Shape Production of Language, not Processingcited by 1
  11. 2025Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domainscited by 0
  12. 2026When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question-Answeringcited by 0
1 more
  1. 2026When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMscited by 0