mlsanityData to 7 Oct 2026

The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning

arXiv 2205.03401 ↗First posted 6 May 2022cs.CLcited by 169

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially. We further show that explanations generated by the LLMs may not entail the models' predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs' predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good--logically consistent with the input and the prediction--more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 46 of its 72 references that are in the map. First posted on arXiv 6 May 2022.

Builds on

46 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
32 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  3. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  6. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  7. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  8. 2022Can language models learn from explanations in context?cited by 248
  9. 2020Selective Question Answering under Domain Shiftcited by 228
  10. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  11. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  12. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  13. 2019Interpretations are useful: penalizing explanations to align neural networks with prior knowledgecited by 144
  14. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  15. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  16. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
  17. 2019Incorporating Priors with Feature Attribution on Text Classificationcited by 68
  18. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  19. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  20. 2018Training Classifiers with Natural Language Explanationscited by 58
  21. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  22. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  23. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  24. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  25. 2022Can Rationalization Improve Robustness?cited by 36
  26. 2019Regularizing Black-box Models for Improved Interpretabilitycited by 35
  27. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  28. 2021Refining Language Models with Compositional Explanationscited by 27
  29. 2021Connecting Attributions and QA Model Behavior on Realistic Counterfactualscited by 20
  30. 2021Can Explanations Be Useful for Calibrating Black Box Models?cited by 20
  31. 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
  32. 2021Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLIcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 50; 2024: 52; 2025: 41; 2026: 17.

169 papers.

Sort:

By area:

  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  2. 2023Steering Language Models With Activation Engineeringcited by 824
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  5. 2023Explainability for Large Language Models: A Surveycited by 331
  6. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  10. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  11. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  12. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
38 more
  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  3. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  4. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  5. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  6. 2023Text Classification via Large Language Modelscited by 104
  7. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  8. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  9. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  10. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  11. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  12. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  13. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  14. 2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
  15. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  16. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  17. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  18. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  19. 2022NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Flycited by 48
  20. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  21. 2024Process-Driven Autoformalization in Lean 4cited by 45
  22. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  23. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  24. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 37
  25. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 31
  26. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  27. 2024AutoPSV: Automated Process-Supervised Verifiercited by 28
  28. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  29. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
  30. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  31. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  32. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  33. 2024OpenTab: Advancing Large Language Models as Open-domain Table Reasonerscited by 25
  34. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  35. 2023Explanation-based Finetuning Makes Models More Robust to Spurious Cuescited by 24
  36. 2024Properties and Challenges of LLM-Generated Explanationscited by 24
  37. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  38. 2022ThinkSum: Probabilistic reasoning over sets using large language modelscited by 23

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
  2. 2023OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models
  3. 2022Explanations from Large Language Models Make Small Reasoners Better
  4. 2023Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
  5. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs
  6. 2023Self-Convinced Prompting: Few-Shot Question Answering with Repeated Introspection
  7. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
  8. 2023Metacognitive Prompting Improves Understanding in Large Language Models