mlsanityData to 13 Oct 2026

Leveraging Large Language Models for Multiple Choice Question Answering

arXiv 2210.12353 ↗First posted 22 Oct 2022cs.CL cs.LGcited by 199

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., "A") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.

Key facts. Cited by 199 papers in the map (data through 13 Oct 2026). Builds on 58 of its 74 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

58 of 74 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2020Scaling Laws for Neural Language Modelscited by 6,494
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,972
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  5. 2022Emergent Abilities of Large Language Modelscited by 2,499
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  8. 2019Language Models as Knowledge Bases?cited by 1,931
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  13. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  14. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,001
  15. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  16. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  17. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 854
  18. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 804
  19. 2018Annotation Artifacts in Natural Language Inference Datacited by 664
  20. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  21. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 536
  22. 2020Scaling Laws for Autoregressive Generative Modelingcited by 514
  23. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  24. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  25. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 414
  26. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 396
  27. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  28. 2021True Few-Shot Learning with Language Modelscited by 343
  29. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  30. 2019Abductive Commonsense Reasoningcited by 300
  31. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 282
  32. 2021Scaling Laws for Transfercited by 268
  33. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 261
  34. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  35. 2022Can large language models reason about medical questions?cited by 194
  36. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  37. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  38. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 46; 2024: 79; 2025: 47; 2026: 27.

199 papers.

Sort:

By area:

  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 470
  2. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 260
  3. 2023DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Textcited by 207
  4. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  5. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 175
  6. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 159
  7. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 145
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  9. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 135
  10. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  11. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 115
38 more
  1. 2023GraphText: Graph Reasoning in Text Spacecited by 110
  2. 2024OLMES: A Standard for Language Model Evaluationscited by 101
  3. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  4. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  6. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 84
  7. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  8. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 76
  9. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  10. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  11. 2023Questioning the Survey Responses of Large Language Modelscited by 73
  12. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 70
  13. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  14. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  15. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  16. 2023Graph Neural Prompting with Large Language Modelscited by 64
  17. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  18. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  19. 2024Language Repository for Long Video Understandingcited by 55
  20. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  21. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 49
  22. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  23. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  24. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  25. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 41
  26. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 40
  27. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 39
  28. 2023Self-Assessment Tests are Unreliable Measures of LLM Personalitycited by 39
  29. 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 39
  30. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  31. 2024FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMscited by 38
  32. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  33. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 32
  34. 2023chatClimate: Grounding Conversational AI in Climate Sciencecited by 32
  35. 2023Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMscited by 32
  36. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  37. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  38. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29

50 most cited of 199 All 199 (JSON)