mlsanityData to 7 Oct 2026

CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms

arXiv 2108.00783 ↗First posted 2 Aug 2021cs.LG cs.AIcited by 85

Not on the map. Why?

Abstract (arXiv)

Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favorable outcomes in the future (e.g., insurance approval). Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available opensource implementations, the decision in favor of certain models is primarily based on what is readily available. Going forward - to guarantee meaningful comparisons across explanation methods - we present CARLA (Counterfactual And Recourse LibrAry), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods. We have open-sourced CARLA and our experimental results on Github, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 38 of its 73 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

38 of 73 references.

References · 38

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  7. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  8. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  9. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  10. 2018Actionable Recourse in Linear Classificationcited by 327
  11. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  12. 2020Algorithmic Recourse: from Counterfactual Explanations to Interventionscited by 250
26 more
  1. 2019Interpretable Counterfactual Explanations Guided by Prototypescited by 223
  2. 2019Model-Agnostic Counterfactual Explanations for Consequential Decisionscited by 204
  3. 2020Multi-Objective Counterfactual Explanationscited by 157
  4. 2018Handling Incomplete Heterogeneous Data using VAEscited by 148
  5. 2019Learning Model-Agnostic Counterfactual Explanations for Tabular Datacited by 146
  6. 2019Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifierscited by 143
  7. 2020Algorithmic recourse under imperfect causal knowledge: a probabilistic approachcited by 139
  8. 2018Explaining Explanations in AIcited by 136
  9. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  10. 2019Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systemscited by 128
  11. 2020A survey of algorithmic recourse: definitions, formulations, solutions, and prospectscited by 118
  12. 2019Efficient Search for Diverse Coherent Explanationscited by 111
  13. 2019The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanationscited by 97
  14. 2020Getting a CLUE: A Method for Explaining Uncertainty Estimatescited by 92
  15. 2017Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweakingcited by 76
  16. 2020On Counterfactual Explanations under Predictive Multiplicitycited by 73
  17. 2020On Generating Plausible Counterfactual and Semi-Factual Explanations for Deep Learningcited by 60
  18. 2019Equalizing Recourse across Groupscited by 51
  19. 2017Inverse Classification for Comparison-based Interpretability in Machine Learningcited by 49
  20. 2020Generative causal explanations of black-box classifierscited by 41
  21. 2021Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertaintiescited by 41
  22. 2014Interpreting Tree Ensembles with inTreescited by 37
  23. 2019PRINCE: Provider-side Interpretability with Counterfactual Explanations in Recommender Systemscited by 35
  24. 2016Making Tree Ensembles Interpretable: A Bayesian Model Selection Approachcited by 27
  25. 2016Generalized Inverse Classificationcited by 22
  26. 2018A Gradient-Based Split Criterion for Highly Accurate and Transparent Model Treescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 17; 2023: 25; 2024: 18; 2025: 9; 2026: 9.

85 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  2. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  3. 2022OpenXAI: Towards a Transparent Evaluation of Model Explanationscited by 125
  4. 2021On the Adversarial Robustness of Causal Algorithmic Recoursecited by 56
  5. 2022Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recoursecited by 40
  6. 2021A Survey on the Robustness of Feature Importance and Counterfactual Explanationscited by 38
  7. 2022On the Privacy Risks of Algorithmic Recoursecited by 32
  8. 2024Robust Counterfactual Explanations in Machine Learning: A Surveycited by 29
  9. 2023GLOBE-CE: A Translation-Based Approach for Global Counterfactual Explanationscited by 27
  10. 2021Counterfactual Shapley Additive Explanationscited by 22
  11. 2022On the Trade-Off between Actionable Explanations and the Right to be Forgottencited by 20
  12. 2023Provably Robust and Plausible Counterfactual Explanations for Neural Networks via Robust Optimisationcited by 20
38 more
  1. 2021Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimatescited by 19
  2. 2022Robust Bayesian Recoursecited by 18
  3. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  4. 2024CEval: A Benchmark for Evaluating Counterfactual Text Generationcited by 15
  5. 2023Generating robust counterfactual explanationscited by 13
  6. 2023Distributionally Robust Recourse Actioncited by 12
  7. 2023Achieving Diversity in Counterfactual Explanations: a Review and Discussioncited by 11
  8. 2021Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functionscited by 10
  9. 2023Towards Bridging the Gaps between the Right to Explanation and the Right to be Forgottencited by 10
  10. 2022Global Counterfactual Explanations: Investigations, Implementations and Improvementscited by 9
  11. 2022Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learningcited by 9
  12. 2022Personalized Algorithmic Recourse with Preference Elicitationcited by 9
  13. 2023Endogenous Macrodynamics in Algorithmic Recoursecited by 9
  14. 2024Counterfactual Explanations via Riemannian Latent Space Traversalcited by 9
  15. 2022Counterfactual Explanations Using Optimization With Constraint Learningcited by 8
  16. 2023CeFlow: A Robust and Efficient Counterfactual Explanation Framework for Tabular Data using Normalizing Flowscited by 8
  17. 2023The Risks of Recourse in Binary Classificationcited by 8
  18. 2024Rethinking Explainable Machine Learning as Applied Statisticscited by 8
  19. 2024TABCF: Counterfactual Explanations for Tabular Data Using a Transformer-Based VAEcited by 8
  20. 2021MCCE: Monte Carlo sampling of realistic counterfactual explanationscited by 7
  21. 2024A multi-criteria approach for selecting an explanation from the set of counterfactuals produced by an ensemble of explainerscited by 7
  22. 2024Interval Abstractions for Robust Counterfactual Explanationscited by 7
  23. 2023Feasible Recourse Plan via Diverse Interpolationcited by 6
  24. 2024Generating Likely Counterfactuals Using Sum-Product Networkscited by 6
  25. 2025RobustX: Robust Counterfactual Explanations Made Easycited by 6
  26. 2023Designing User-Centric Behavioral Interventions to Prevent Dysglycemia with Novel Counterfactual Explanationscited by 5
  27. 2023Privacy-Preserving Algorithmic Recoursecited by 5
  28. 2023Faithful Model Explanations through Energy-Constrained Conformal Counterfactualscited by 5
  29. 2025From Search To Sampling: Generative Models For Robust Algorithmic Recoursecited by 5
  30. 2022Standardized Evaluation of Machine Learning Methods for Evolving Data Streamscited by 4
  31. 2022Computing Rule-Based Explanations by Leveraging Counterfactualscited by 4
  32. 2023Explaining Black-Box Models through Counterfactualscited by 4
  33. 2024Generating Feasible and Plausible Counterfactual Explanations for Outcome Prediction of Business Processescited by 4
  34. 2024Enhancing Counterfactual Explanation Search with Diffusion Distance and Directional Coherencecited by 4
  35. 2026What-If Explanations Over Time: Counterfactuals for Time Series Classificationcited by 4
  36. 2022TSInterpret: A unified framework for time series interpretabilitycited by 3
  37. 2023Explainable Predictive Maintenancecited by 3
  38. 2023XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classificationcited by 3

50 most cited of 85 All 85 (JSON)