mlsanityData to 7 Oct 2026

ERASER: A Benchmark to Evaluate Rationalized NLP Models

arXiv 1911.03429 ↗First posted 8 Nov 2019cs.CL cs.AI cs.LGcited by 523

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the `reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of "rationales" (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/

Key facts. Cited by 523 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 81; 2022: 94; 2023: 86; 2024: 81; 2025: 74; 2026: 107.

523 papers.

Sort:

By area:

  1. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  2. 2021A Survey on Automated Fact-Checkingcited by 376
  3. 2023Explainability for Large Language Models: A Surveycited by 331
  4. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  5. 2021Trustworthy AI: From Principles to Practicescited by 206
  6. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  7. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2021Explaining Answers with Entailment Treescited by 148
  10. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  11. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  12. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
38 more
  1. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  2. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  3. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  4. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  5. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  6. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  7. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  8. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  9. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  10. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  11. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  12. 2021Synthetic Benchmarks for Scientific Research in Explainable Machine Learningcited by 59
  13. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  14. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  15. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  16. 2024Learning to Compress Prompt in Natural Language Formatscited by 54
  17. 2021ProoFVer: Natural Logic Theorem Proving for Fact Verificationcited by 52
  18. 2023Sequential Integrated Gradients: a simple but effective method for explaining language modelscited by 52
  19. 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 49
  20. 2022e-CARE: a New Dataset for Exploring Explainable Causal Reasoningcited by 47
  21. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  22. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  23. 2021Discretized Integrated Gradients for Explaining Language Modelscited by 45
  24. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  25. 2021SelfExplain: A Self-Explaining Architecture for Neural Text Classifierscited by 44
  26. 2025Walk the Talk? Measuring the Faithfulness of Large Language Model Explanationscited by 44
  27. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  28. 2023Quantifying Context Mixing in Transformerscited by 43
  29. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 38
  30. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  31. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  32. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  33. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  34. 2021Probing GNN Explainers: A Rigorous Theoretical and Empirical Analysis of GNN Explanation Methodscited by 36
  35. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  36. 2022Can Rationalization Improve Robustness?cited by 36
  37. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 35
  38. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35

50 most cited of 523 All 523 (JSON)