# When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data

arXiv: https://arxiv.org/abs/2102.02201 · HTML: https://mlsanity.com/paper/2102.02201/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 3 Feb 2021.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [WT5?! Training Text-to-Text Models to Explain their Predictions](https://mlsanity.com/paper/2004.14546/) (2020)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations](https://mlsanity.com/paper/2011.12854/) (2020)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [Learning with Latent Language](https://mlsanity.com/paper/1711.00482/) (2017)
- [Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions](https://mlsanity.com/paper/1506.00511/) (2015)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)

## Built on by (most cited first, 30 of 75)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [Cumulative Reasoning with Large Language Models](https://mlsanity.com/paper/2308.04371/) (2023)
- [Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs](https://mlsanity.com/paper/2111.13654/) (2021)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2402.04616/) (2024)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization](https://mlsanity.com/paper/2411.06208/) (2024)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [MCC-KD: Multi-CoT Consistent Knowledge Distillation](https://mlsanity.com/paper/2310.14747/) (2023)
- [What to Learn, and How: Toward Effective Learning from Rationales](https://mlsanity.com/paper/2112.00071/) (2021)
- [AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation](https://mlsanity.com/paper/2504.07532/) (2025)
- [InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis](https://mlsanity.com/paper/2302.08624/) (2023)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning](https://mlsanity.com/paper/2312.05762/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- ["Why is this misleading?": Detecting News Headline Hallucinations with Explanations](https://mlsanity.com/paper/2302.05852/) (2023)
- [How Many Data Samples is an Additional Instruction Worth?](https://mlsanity.com/paper/2203.09161/) (2022)

## Abstract (arXiv)

Many methods now exist for conditioning model outputs on task instructions, retrieved documents, and user-provided explanations and feedback. Rather than relying solely on examples of task inputs and outputs, these approaches use valuable additional data for improving model correctness and aligning learned models with human priors. Meanwhile, a growing body of evidence suggests that some language models can (1) store a large amount of knowledge in their parameters, and (2) perform inference over tasks in textual inputs at test time. These results raise the possibility that, for some tasks, humans cannot explain to a model any more about the task than it already knows or could infer on its own. In this paper, we study the circumstances under which explanations of individual data points can (or cannot) improve modeling performance. In order to carefully control important properties of the data and explanations, we introduce a synthetic dataset for experiments, and we also make use of three existing datasets with explanations: e-SNLI, TACRED, and SemEval. We first give a formal framework for the available modeling approaches, in which explanation data can be used as model inputs, as targets, or as a prior. After arguing that the most promising role for explanation data is as model inputs, we propose to use a retrieval-based method and show that it solves our synthetic task with accuracies upwards of 95%, while baselines without explanation data achieve below 65% accuracy. We then identify properties of datasets for which retrieval-based modeling fails. With the three existing datasets, we find no improvements from explanation retrieval. Drawing on findings from our synthetic task, we suggest that at least one of six preconditions for successful modeling fails to hold with these datasets. Our code is publicly available at https://github.com/peterbhase/ExplanationRoles