# LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study

arXiv: https://arxiv.org/abs/2405.00722 · HTML: https://mlsanity.com/paper/2405.00722/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Towards LLM-guided Causal Explainability for Black-box Text Classifiers](https://mlsanity.com/paper/2309.13340/) (2023)
- [Prompting Large Language Models for Counterfactual Generation: An Empirical Study](https://mlsanity.com/paper/2305.14791/) (2023)
- [Benchmarking Large Language Model Capabilities for Conditional Generation](https://mlsanity.com/paper/2306.16793/) (2023)
- [CREST: A Joint Framework for Rationalization and Counterfactual Text Generation](https://mlsanity.com/paper/2305.17075/) (2023)
- [How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?](https://mlsanity.com/paper/2109.07022/) (2021)
- [CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration](https://mlsanity.com/paper/2309.07822/) (2023)

## Built on by (most cited first, 30 of 31)

- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [A Survey on Natural Language Counterfactual Generation](https://mlsanity.com/paper/2407.03993/) (2024)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs](https://mlsanity.com/paper/2406.11514/) (2024)
- [LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations](https://mlsanity.com/paper/2509.09396/) (2025)
- [Counterfactual Token Generation in Large Language Models](https://mlsanity.com/paper/2409.17027/) (2024)
- [Reasoning Elicitation in Language Models via Counterfactual Feedback](https://mlsanity.com/paper/2410.03767/) (2024)
- [Can LLMs Explain Themselves Counterfactually?](https://mlsanity.com/paper/2502.18156/) (2025)
- [Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code](https://mlsanity.com/paper/2510.01539/) (2025)
- [Interpreting Language Reward Models via Contrastive Explanations](https://mlsanity.com/paper/2411.16502/) (2024)
- [Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic](https://mlsanity.com/paper/2504.05262/) (2025)
- [Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations](https://mlsanity.com/paper/2601.03775/) (2026)
- [FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation](https://mlsanity.com/paper/2501.00777/) (2025)
- [On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study](https://mlsanity.com/paper/2505.11839/) (2025)
- [Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals](https://mlsanity.com/paper/2505.13972/) (2025)
- [Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanations](https://mlsanity.com/paper/2601.00282/) (2026)
- [Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations](https://mlsanity.com/paper/2510.21631/) (2025)
- [Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models](https://mlsanity.com/paper/2603.16600/) (2026)
- [Guiding LLMs to Generate High-Fidelity and High-Quality Counterfactual Explanations for Text Classification](https://mlsanity.com/paper/2503.04463/) (2025)
- [The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference](https://mlsanity.com/paper/2508.10777/) (2025)
- [Enhancing XAI Narratives through Multi-Narrative Refinement and Knowledge Distillation](https://mlsanity.com/paper/2510.03134/) (2025)
- [SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits](https://mlsanity.com/paper/2412.13378/) (2024)
- [Deriving Strategic Market Insights with Large Language Models: A Benchmark for Forward Counterfactual Generation](https://mlsanity.com/paper/2505.19430/) (2025)
- [iFlip: Iterative Feedback-driven Counterfactual Example Refinement](https://mlsanity.com/paper/2601.01446/) (2026)
- [DRIV-EX: Counterfactual Explanations for Driving LLMs](https://mlsanity.com/paper/2603.00696/) (2026)
- [Refining Financial Consumer Complaints through Multi-Scale Model Interaction](https://mlsanity.com/paper/2504.09903/) (2025)
- [Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forward](https://mlsanity.com/paper/2601.13122/) (2026)
- [Gumbel Machine: Counterfactual Student Writing Generation via Gumbel Noise Steering](https://mlsanity.com/paper/2605.27249/) (2026)
- [CoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction](https://mlsanity.com/paper/2606.15069/) (2026)
- [Taking the Whys Seriously: Limitations of Counterfactual Explanations in Justification and Recourse](https://mlsanity.com/paper/2608.30956/) (2026)

## Abstract (arXiv)

As NLP models become more complex, understanding their decisions becomes more crucial. Counterfactuals (CFs), where minimal changes to inputs flip a model's prediction, offer a way to explain these models. While Large Language Models (LLMs) have shown remarkable performance in NLP tasks, their efficacy in generating high-quality CFs remains uncertain. This work fills this gap by investigating how well LLMs generate CFs for two NLU tasks. We conduct a comprehensive comparison of several common LLMs, and evaluate their CFs, assessing both intrinsic metrics, and the impact of these CFs on data augmentation. Moreover, we analyze differences between human and LLM-generated CFs, providing insights for future research directions. Our results show that LLMs generate fluent CFs, but struggle to keep the induced changes minimal. Generating CFs for Sentiment Analysis (SA) is less challenging than NLI where LLMs show weaknesses in generating CFs that flip the original label. This also reflects on the data augmentation performance, where we observe a large gap between augmenting with human and LLMs CFs. Furthermore, we evaluate LLMs' ability to assess CFs in a mislabelled data setting, and show that they have a strong bias towards agreeing with the provided labels. GPT4 is more robust against this bias and its scores correlate well with automatic metrics. Our findings reveal several limitations and point to potential future work directions.