# Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments

arXiv: https://arxiv.org/abs/2410.21131 · HTML: https://mlsanity.com/paper/2410.21131/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [The Measure and Mismeasure of Fairness](https://mlsanity.com/paper/1808.00023/) (2018)
- [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI](https://mlsanity.com/paper/1902.01876/) (2019)
- [Towards LLM-guided Causal Explainability for Black-box Text Classifiers](https://mlsanity.com/paper/2309.13340/) (2023)
- [PRINCE: Provider-side Interpretability with Counterfactual Explanations in Recommender Systems](https://mlsanity.com/paper/1911.08378/) (2019)
- [A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models](https://mlsanity.com/paper/1804.02969/) (2018)
- [CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models](https://mlsanity.com/paper/2109.01401/) (2021)
- [CARE: Coherent Actionable Recourse based on Sound Counterfactual Explanations](https://mlsanity.com/paper/2108.08197/) (2021)
- [Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI](https://mlsanity.com/paper/2204.10152/) (2022)
- [Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction](https://mlsanity.com/paper/2407.13166/) (2024)
- [Understanding Consumer Preferences for Explanations Generated by XAI Algorithms](https://mlsanity.com/paper/2107.02624/) (2021)
- [CountARFactuals -- Generating plausible model-agnostic counterfactual explanations with adversarial random forests](https://mlsanity.com/paper/2404.03506/) (2024)
- [Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systems](https://mlsanity.com/paper/2405.13362/) (2024)
- [For Better or Worse: The Impact of Counterfactual Explanations' Directionality on User Behavior in xAI](https://mlsanity.com/paper/2306.07637/) (2023)
- [Enhancing Counterfactual Explanation Search with Diffusion Distance and Directional Coherence](https://mlsanity.com/paper/2404.12810/) (2024)

## Built on by (most cited first, 8 of 8)

- [FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation](https://mlsanity.com/paper/2501.00777/) (2025)
- [Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals](https://mlsanity.com/paper/2505.13972/) (2025)
- [Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanations](https://mlsanity.com/paper/2601.00282/) (2026)
- [Predicting Satisfaction of Counterfactual Explanations from Human Ratings of Explanatory Qualities](https://mlsanity.com/paper/2504.13899/) (2025)
- [Knowing What You Cannot Explain: Learning to Reject Low-Quality Explanations](https://mlsanity.com/paper/2507.12900/) (2025)
- [Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation](https://mlsanity.com/paper/2601.00263/) (2026)
- [iFlip: Iterative Feedback-driven Counterfactual Example Refinement](https://mlsanity.com/paper/2601.01446/) (2026)
- [On the Importance and Evaluation of Narrativity in Natural Language AI Explanations](https://mlsanity.com/paper/2604.18311/) (2026)

## Abstract (arXiv)

As machine learning models evolve, maintaining transparency demands more human-centric explainable AI techniques. Counterfactual explanations, with roots in human reasoning, identify the minimal input changes needed to obtain a given output and, hence, are crucial for supporting decision-making. Despite their importance, the evaluation of these explanations often lacks grounding in user studies and remains fragmented, with existing metrics not fully capturing human perspectives. To address this challenge, we developed a diverse set of 30 counterfactual scenarios and collected ratings across 8 evaluation metrics from 206 respondents. Subsequently, we fine-tuned different Large Language Models (LLMs) to predict average or individual human judgment across these metrics. Our methodology allowed LLMs to achieve an accuracy of up to 63% in zero-shot evaluations and 85% (over a 3-classes prediction) with fine-tuning across all metrics. The fine-tuned models predicting human ratings offer better comparability and scalability in evaluating different counterfactual explanation frameworks.