# REV: Information-Theoretic Evaluation of Free-Text Rationales

arXiv: https://arxiv.org/abs/2210.04982 · HTML: https://mlsanity.com/paper/2210.04982/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [WT5?! Training Text-to-Text Models to Explain their Predictions](https://mlsanity.com/paper/2004.14546/) (2020)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Learning to Faithfully Rationalize by Construction](https://mlsanity.com/paper/2005.00115/) (2020)
- [Restricting the Flow: Information Bottlenecks for Attribution](https://mlsanity.com/paper/2001.00396/) (2020)
- [Transforming Question Answering Datasets Into Natural Language Inference Datasets](https://mlsanity.com/paper/1809.02922/) (2018)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)
- [QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions](https://mlsanity.com/paper/1909.03553/) (2019)

## Built on by (most cited first, 22 of 22)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://mlsanity.com/paper/2304.10703/) (2023)
- [Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales](https://mlsanity.com/paper/2305.07095/) (2023)
- [SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2410.08474/) (2024)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach](https://mlsanity.com/paper/2502.00577/) (2025)
- [Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods](https://mlsanity.com/paper/2210.07222/) (2022)
- [CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification](https://mlsanity.com/paper/2303.03628/) (2023)
- [Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning](https://mlsanity.com/paper/2311.07099/) (2023)
- [RORA: Robust Free-Text Rationale Evaluation](https://mlsanity.com/paper/2402.18678/) (2024)
- [Rethinking Human Preference Evaluation of LLM Rationales](https://mlsanity.com/paper/2509.11026/) (2025)
- [Situated Natural Language Explanations](https://mlsanity.com/paper/2308.14115/) (2023)
- [Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability](https://mlsanity.com/paper/2504.16056/) (2025)
- [KNIFE: Distilling Reasoning Knowledge From Free-Text Rationales](https://mlsanity.com/paper/2212.09721/) (2022)
- [Measuring Information in Text Explanations](https://mlsanity.com/paper/2310.04557/) (2023)
- [TVE: Learning Meta-attribution for Transferable Vision Explainer](https://mlsanity.com/paper/2312.15359/) (2023)
- [Agentic Multi-Persona Framework for Evidence-Aware Fake News Detection](https://mlsanity.com/paper/2512.21039/) (2025)
- [Measuring Pointwise V-Usable Information In-Context-ly](https://mlsanity.com/paper/2310.12300/) (2023)
- [Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations](https://mlsanity.com/paper/2509.25844/) (2025)
- [Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain](https://mlsanity.com/paper/2603.17815/) (2026)

## Abstract (arXiv)

Generating free-text rationales is a promising step towards explainable NLP, yet evaluating such rationales remains a challenge. Existing metrics have mostly focused on measuring the association between the rationale and a given label. We argue that an ideal metric should focus on the new information uniquely provided in the rationale that is otherwise not provided in the input or the label. We investigate this research problem from an information-theoretic perspective using conditional V-information (Hewitt et al., 2021). More concretely, we propose a metric called REV (Rationale Evaluation with conditional V-information), to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label. Experiments across four benchmarks with reasoning tasks, including chain-of-thought, demonstrate the effectiveness of REV in evaluating rationale-label pairs, compared to existing metrics. We further demonstrate REV is consistent with human judgments on rationale evaluations and provides more sensitive measurements of new information in free-text rationales. When used alongside traditional performance metrics, REV provides deeper insights into models' reasoning and prediction processes.