# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data

arXiv: https://arxiv.org/abs/2107.04680 · HTML: https://mlsanity.com/paper/2107.04680/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 89 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 34)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Algorithmic Recourse: from Counterfactual Explanations to Interventions](https://mlsanity.com/paper/2002.06278/) (2020)
- [Interpretable Counterfactual Explanations Guided by Prototypes](https://mlsanity.com/paper/1907.02584/) (2019)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Model-Agnostic Counterfactual Explanations for Consequential Decisions](https://mlsanity.com/paper/1905.11190/) (2019)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers](https://mlsanity.com/paper/1912.03277/) (2019)
- [A survey of algorithmic recourse: definitions, formulations, solutions, and prospects](https://mlsanity.com/paper/2010.04050/) (2020)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021)
- [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020)
- [Robust Counterfactual Explanations on Graph Neural Networks](https://mlsanity.com/paper/2107.04086/) (2021)
- [Inverse Classification for Comparison-based Interpretability in Machine Learning](https://mlsanity.com/paper/1712.08443/) (2017)
- [Measurable Counterfactual Local Explanations for Any Classifier](https://mlsanity.com/paper/1908.03020/) (2019)
- [FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles](https://mlsanity.com/paper/1911.12199/) (2019)
- [CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models](https://mlsanity.com/paper/1905.07857/) (2019)
- [Convex Density Constraints for Computing Plausible Counterfactual Explanations](https://mlsanity.com/paper/2002.04862/) (2020)
- [Counterfactual Explanation Algorithms for Behavioral and Textual Data](https://mlsanity.com/paper/1912.01819/) (2019)
- [Explaining Data-Driven Decisions made by AI Systems: The Counterfactual Approach](https://mlsanity.com/paper/2001.07417/) (2020)
- [BRPO: Batch Residual Policy Optimization](https://mlsanity.com/paper/2002.05522/) (2020)
- [Generating Counterfactual and Contrastive Explanations using SHAP](https://mlsanity.com/paper/1906.09293/) (2019)
- [Synthesizing Action Sequences for Modifying Model Decisions](https://mlsanity.com/paper/1910.00057/) (2019)
- [Explaining Deep Learning Models with Constrained Adversarial Examples](https://mlsanity.com/paper/1906.10671/) (2019)

## Built on by (most cited first, 19 of 19)

- [The privacy issue of counterfactual explanations: explanation linkage attacks](https://mlsanity.com/paper/2210.12051/) (2022)
- [VCNet: A self-explaining model for realistic counterfactual generation](https://mlsanity.com/paper/2212.10847/) (2022)
- [Generating robust counterfactual explanations](https://mlsanity.com/paper/2304.12943/) (2023)
- [Keep Your Friends Close and Your Counterfactuals Closer: Improved Learning From Closest Rather Than Plausible Counterfactual Explanations in an Abstract Setting](https://mlsanity.com/paper/2205.05515/) (2022)
- [Achieving Diversity in Counterfactual Explanations: a Review and Discussion](https://mlsanity.com/paper/2305.05840/) (2023)
- [Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learning](https://mlsanity.com/paper/2205.03398/) (2022)
- [Endogenous Macrodynamics in Algorithmic Recourse](https://mlsanity.com/paper/2308.08187/) (2023)
- [TIGTEC : Token Importance Guided TExt Counterfactuals](https://mlsanity.com/paper/2304.12425/) (2023)
- [Benchmarking Instance-Centric Counterfactual Algorithms for XAI: From White Box to Black Box](https://mlsanity.com/paper/2203.02399/) (2022)
- [Manipulation Risks in Explainable AI: The Implications of the Disagreement Problem](https://mlsanity.com/paper/2306.13885/) (2023)
- [Designing User-Centric Behavioral Interventions to Prevent Dysglycemia with Novel Counterfactual Explanations](https://mlsanity.com/paper/2310.01684/) (2023)
- [counterfactuals: An R Package for Counterfactual Explanation Methods](https://mlsanity.com/paper/2304.06569/) (2023)
- [Unveiling the Potential of Counterfactuals Explanations in Employability](https://mlsanity.com/paper/2305.10069/) (2023)
- [Calculating and Visualizing Counterfactual Feature Importance Values](https://mlsanity.com/paper/2306.06506/) (2023)
- [Mitigating Text Toxicity with Counterfactual Generation](https://mlsanity.com/paper/2405.09948/) (2024)
- [Adaptive Adversarial Training Does Not Increase Recourse Costs](https://mlsanity.com/paper/2309.02528/) (2023)
- [On the Definition and Detection of Cherry-Picking in Counterfactual Explanations](https://mlsanity.com/paper/2601.04977/) (2026)
- [Physics-Guided Counterfactual Explanations for Large-Scale Multivariate Time Series: Application in Scalable and Interpretable SEP Event Prediction](https://mlsanity.com/paper/2601.08999/) (2026)
- [CEL: Comprehensive Counterfactual Explanations Library and Benchmark](https://mlsanity.com/paper/2607.22045/) (2026)

## Abstract (arXiv)

Counterfactual explanations are viewed as an effective way to explain machine learning predictions. This interest is reflected by a relatively young literature with already dozens of algorithms aiming to generate such explanations. These algorithms are focused on finding how features can be modified to change the output classification. However, this rather general objective can be achieved in different ways, which brings about the need for a methodology to test and benchmark these algorithms. The contributions of this work are manifold: First, a large benchmarking study of 10 algorithmic approaches on 22 tabular datasets is performed, using 9 relevant evaluation metrics. Second, the introduction of a novel, first of its kind, framework to test counterfactual generation algorithms. Third, a set of objective metrics to evaluate and compare counterfactual results. And finally, insight from the benchmarking results that indicate which approaches obtain the best performance on what type of dataset. This benchmarking study and framework can help practitioners in determining which technique and building blocks most suit their context, and can help researchers in the design and evaluation of current and future counterfactual generation algorithms. Our findings show that, overall, there's no single best algorithm to generate counterfactual explanations as the performance highly depends on properties related to the dataset, model, score and factual point specificities.