mlsanityData to 7 Oct 2026

The Irrationality of Neural Rationale Models

arXiv 2110.07550 ↗First posted 14 Oct 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Neural rationale models are popular for interpretable predictions of NLP tasks. In these, a selector extracts segments of the input text, called rationales, and passes these segments to a classifier for prediction. Since the rationale is the only information accessible to the classifier, it is plausibly defined as the explanation. Is such a characterization unconditionally correct? In this paper, we argue to the contrary, with both philosophical perspectives and empirical evidence suggesting that rationale models are, perhaps, less rational and interpretable than expected. We call for more rigorous and comprehensive evaluations of these models to ensure desired properties of interpretability are indeed achieved. The code can be found at https://github.com/yimingz89/Neural-Rationale-Analysis.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

20 of 29 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2018Sanity Checks for Saliency Mapscited by 1,149
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2019Attention is not Explanationcited by 916
  10. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  11. 2017Interpretation of Neural Networks is Fragilecited by 443
  12. 2018On the Robustness of Interpretability Methodscited by 310
8 more
  1. 2016Rationalizing Neural Predictionscited by 281
  2. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  3. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  4. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  5. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  6. 2019Rethinking Cooperative Rationalization: Introspective Extraction and Complement Controlcited by 96
  7. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  8. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 5; 2024: 3; 2025: 2.

16 papers.

Sort:

By area:

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  3. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 66
  4. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  5. 2023D-Separation for Causal Self-Explanationcited by 21
  6. 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
  7. 2024F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AIcited by 17
  8. 2022ExSum: From Local Explanations to Model Understandingcited by 15
  9. 2023Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraintcited by 14
  10. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  11. 2024Intrinsic User-Centric Interpretability through Global Mixture of Expertscited by 9
  12. 2025Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasetscited by 6
4 more
  1. 2025B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainabilitycited by 4
  2. 2023Unsupervised Selective Rationalization with Noise Injectioncited by 3
  3. 2022SparCAssist: A Model Risk Assessment Assistant Based on Sparse Generated Counterfactualscited by 1
  4. 2024Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictionscited by 1