mlsanityData to 14 Oct 2026

Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions

arXiv 2509.23782 ↗First posted 28 Sep 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) perform strongly on diverse tasks, their trustworthiness is limited by erratic behavior that is unfaithful to their internal knowledge. In particular, LLMs often fail on multiple-choice questions (MCQs) even if they encode correct answers in their hidden representations, revealing a misalignment between internal knowledge and output behavior. We investigate and mitigate this knowledge-prediction gap on MCQs through a three-step analysis of hidden representations. First, we quantify the prevalence and magnitude of the gap across models and datasets. Second, we provide a geometric interpretation by identifying distinct knowledge and prediction subspaces in the residual stream. Third, we introduce KAPPA, a lightweight inference-time intervention that aligns the two subspaces within the residual stream to reduce the knowledge-prediction gap. Our results provide a geometric and interpretable explanation of the knowledge-prediction gap in LLMs. Furthermore, KAPPA effectively reduces the gap across diverse MCQ benchmarks and models, and generalizes to free-form settings.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 58 of its 94 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

58 of 94 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
38 more
  1. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  13. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  14. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  15. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  16. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  17. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  18. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  19. 2023In-Context Learning Creates Task Vectorscited by 312
  20. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  21. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  22. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  23. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  24. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  25. 2025Reasoning Models Can Be Effective Without Thinkingcited by 187
  26. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  27. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  28. 2024Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Modelscited by 86
  29. 2023Alignment for Honestycited by 74
  30. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  31. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  32. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  33. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 61
  34. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  35. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  36. 2024A Language Model's Guide Through Latent Spacecited by 43
  37. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  38. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 40

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMscited by 2
  2. 2026Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Modelscited by 1
  3. 2026Legible Failures: Detecting and Repairing In-Context Binding Errorscited by 1
  4. 2026The Low-Rank Structure of VLA Reinforcement Learningcited by 0
  5. 2026Readout Stability in Prefill-Only Decision Models:Zero-Label Prediction and Inference-Time Compute Allocationcited by 0
  6. 2026FunDM: Cheap & Label-Free Decision Models from Frozen Language Modelscited by 0