# Learning to Scaffold: Optimizing Model Explanations for Teaching

arXiv: https://arxiv.org/abs/2204.10810 · HTML: https://mlsanity.com/paper/2204.10810/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 76 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)

## Built on by (most cited first, 20 of 20)

- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [D-Separation for Causal Self-Explanation](https://mlsanity.com/paper/2309.13391/) (2023)
- [Studying How to Efficiently and Effectively Guide Models with Explanations](https://mlsanity.com/paper/2303.11932/) (2023)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)
- [Enhancing the Rationale-Input Alignment for Self-explaining Rationalization](https://mlsanity.com/paper/2312.04103/) (2023)
- [Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint](https://mlsanity.com/paper/2305.13599/) (2023)
- [Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models](https://mlsanity.com/paper/2208.11445/) (2022)
- [Explainable Bayesian Optimization](https://mlsanity.com/paper/2401.13334/) (2024)
- [MEGAN: Multi-Explanation Graph Attention Network](https://mlsanity.com/paper/2211.13236/) (2022)
- [Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models](https://mlsanity.com/paper/2311.01732/) (2023)
- [Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization](https://mlsanity.com/paper/2503.06202/) (2025)
- [Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Framework](https://mlsanity.com/paper/2406.15085/) (2024)
- [Explanation Regeneration via Information Bottleneck](https://mlsanity.com/paper/2212.09603/) (2022)
- [ExaRanker: Explanation-Augmented Neural Ranker](https://mlsanity.com/paper/2301.10521/) (2023)
- [FastRM: An efficient and automatic explainability framework for multimodal generative models](https://mlsanity.com/paper/2412.01487/) (2024)
- [Quantifying the Intrinsic Usefulness of Attributional Explanations for Graph Neural Networks with Artificial Simulatability Studies](https://mlsanity.com/paper/2305.15961/) (2023)
- [Evaluating Saliency Explanations in NLP by Crowdsourcing](https://mlsanity.com/paper/2405.10767/) (2024)
- [Explanation Regularisation through the Lens of Attributions](https://mlsanity.com/paper/2407.16693/) (2024)
- [Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index](https://mlsanity.com/paper/2608.05411/) (2026)

## Abstract (arXiv)

Modern machine learning models are opaque, and as a result there is a burgeoning academic subfield on methods that explain these models' behavior. However, what is the precise goal of providing such explanations, and how can we demonstrate that explanations achieve this goal? Some research argues that explanations should help teach a student (either human or machine) to simulate the model being explained, and that the quality of explanations can be measured by the simulation accuracy of students on unexplained examples. In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model. We train models on three natural language processing and computer vision tasks, and find that students trained with explanations extracted with our framework are able to simulate the teacher significantly more effectively than ones produced with previous methods. Through human annotations and a user study, we further find that these learned explanations more closely align with how humans would explain the required decisions in these tasks. Our code is available at https://github.com/coderpat/learning-scaffold