# Explain and Predict, and then Predict Again

arXiv: https://arxiv.org/abs/2101.04109 · HTML: https://mlsanity.com/paper/2101.04109/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Interpretable Neural Predictions with Differentiable Binary Variables](https://mlsanity.com/paper/1905.08160/) (2019)
- [An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1902.00006/) (2019)
- [Inferring Which Medical Treatments Work from Reports of Clinical Trials](https://mlsanity.com/paper/1904.01606/) (2019)
- [Rationale-Augmented Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/1605.04469/) (2016)
- [Do Human Rationales Improve Machine Explanations?](https://mlsanity.com/paper/1905.13714/) (2019)
- [Attention-over-Attention Neural Networks for Reading Comprehension](https://mlsanity.com/paper/1607.04423/) (2016)
- [Fine-grained Sentiment Analysis with Faithful Attention](https://mlsanity.com/paper/1908.06870/) (2019)
- [TableNet: An Approach for Determining Fine-grained Relations for Wikipedia Tables](https://mlsanity.com/paper/1902.01740/) (2019)
- [Expedition: A Time-Aware Exploratory Search System Designed for Scholars](https://mlsanity.com/paper/1810.10769/) (2018)
- [The Dawn of Today's Popular Domains: A Study of the Archived German Web over 18 Years](https://mlsanity.com/paper/1702.01151/) (2017)

## Built on by (most cited first, 29 of 29)

- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [BERTnesia: Investigating the capture and forgetting of knowledge in BERT](https://mlsanity.com/paper/2106.02902/) (2021)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Rationalization for Explainable NLP: A Survey](https://mlsanity.com/paper/2301.08912/) (2023)
- [SPECTRA: Sparse Structured Text Rationalization](https://mlsanity.com/paper/2109.04552/) (2021)
- [Learning by Self-Explaining](https://mlsanity.com/paper/2309.08395/) (2023)
- [CAVES: A Dataset to facilitate Explainable Classification and Summarization of Concerns towards COVID Vaccines](https://mlsanity.com/paper/2204.13746/) (2022)
- [Sparse and Structured Hopfield Networks](https://mlsanity.com/paper/2402.13725/) (2024)
- [Evaluating Generative Ad Hoc Information Retrieval](https://mlsanity.com/paper/2311.04694/) (2023)
- [Query Understanding in the Age of Large Language Models](https://mlsanity.com/paper/2306.16004/) (2023)
- [Power of Explanations: Towards automatic debiasing in hate speech detection](https://mlsanity.com/paper/2209.09975/) (2022)
- [Exploring the Trade-off Between Model Performance and Explanation Plausibility of Text Classifiers Using Human Rationales](https://mlsanity.com/paper/2404.03098/) (2024)
- [Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs](https://mlsanity.com/paper/2502.08909/) (2025)
- [Learnt Sparsification for Interpretable Graph Neural Networks](https://mlsanity.com/paper/2106.12920/) (2021)
- [FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn Correction in the Loop](https://mlsanity.com/paper/2110.10144/) (2021)
- [Rationale-Guided Few-Shot Classification to Detect Abusive Language](https://mlsanity.com/paper/2211.17046/) (2022)
- [ExClaim: Explainable Neural Claim Verification Using Rationalization](https://mlsanity.com/paper/2301.08914/) (2023)
- [RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank](https://mlsanity.com/paper/2405.01848/) (2024)
- [Consistent Multi-Granular Rationale Extraction for Explainable Multi-hop Fact Verification](https://mlsanity.com/paper/2305.09400/) (2023)
- [Dual-Head Reasoning Distillation: Improving Classifier Accuracy with Train-Time-Only Reasoning](https://mlsanity.com/paper/2509.21487/) (2025)
- [SparCAssist: A Model Risk Assessment Assistant Based on Sparse Generated Counterfactuals](https://mlsanity.com/paper/2205.01588/) (2022)
- [Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictions](https://mlsanity.com/paper/2407.11778/) (2024)
- [Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions](https://mlsanity.com/paper/2202.01096/) (2022)
- [DISCO: DISCovering Overfittings as Causal Rules for Text Classification Models](https://mlsanity.com/paper/2411.04649/) (2024)
- [Can human clinical rationales improve the performance and explainability of clinical text classification models?](https://mlsanity.com/paper/2507.21302/) (2025)
- [KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning](https://mlsanity.com/paper/2509.15676/) (2025)
- [CREST: Improving Interpretability and Effectiveness of Troubleshooting at Ericsson through Criterion-Specific Trouble Report Retrieval](https://mlsanity.com/paper/2511.17417/) (2025)
- [Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media](https://mlsanity.com/paper/2603.18611/) (2026)

## Abstract (arXiv)

A desirable property of learning systems is to be both effective and interpretable. Towards this goal, recent models have been proposed that first generate an extractive explanation from the input text and then generate a prediction on just the explanation called explain-then-predict models. These models primarily consider the task input as a supervision signal in learning an extractive explanation and do not effectively integrate rationales data as an additional inductive bias to improve task performance. We propose a novel yet simple approach ExPred, that uses multi-task learning in the explanation generation phase effectively trading-off explanation and prediction losses. And then we use another prediction network on just the extracted explanations for optimizing the task performance. We conduct an extensive evaluation of our approach on three diverse language datasets -- fact verification, sentiment classification, and QA -- and find that we substantially outperform existing approaches.