mlsanityData to 14 Oct 2026

CEval: A Benchmark for Evaluating Counterfactual Text Generation

arXiv 2404.17475 ↗First posted 26 Apr 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Counterfactual text generation aims to minimally change a text, such that it is classified differently. Judging advancements in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our experiments found no perfect method for generating counterfactual text. Methods that excel at counterfactual metrics often produce lower-quality text while LLMs with simple prompts generate high-quality text but struggle with counterfactual criteria. By making CEval available as an open-source Python library, we encourage the community to contribute more methods and maintain consistent evaluation in future work.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

33 of 51 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  12. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
21 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  3. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  4. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  7. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  8. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  9. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  10. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 106
  11. 2021CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithmscited by 86
  12. 2020Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Textcited by 75
  13. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 68
  14. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  15. 2023Towards LLM-guided Causal Explainability for Black-box Text Classifierscited by 44
  16. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  17. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 37
  18. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  19. 2022CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generationcited by 24
  20. 2023CREST: A Joint Framework for Rationalization and Counterfactual Text Generationcited by 24
  21. 2021SPECTRA: Sparse Structured Text Rationalizationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  2. 2025What Has Been Lost with Synthetic Evaluation?cited by 9
  3. 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7
  4. 2025Merge-of-Thought Distillationcited by 7
  5. 2025Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactualscited by 5
  6. 2026Training-Trajectory-Aware Token Selectioncited by 4
  7. 2025Guiding LLMs to Generate High-Fidelity and High-Quality Counterfactual Explanations for Text Classificationcited by 3
  8. 2025WeMusic-Agent: Efficient Conversational Music Recommendation via Knowledge Internalization and Agentic Boundary Learningcited by 2
  9. 2025Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactualscited by 1
  10. 2025Conceptual Contrastive Edits in Textual and Vision-Language Retrievalcited by 0
  11. 2025PsyLite Technical Reportcited by 0
  12. 2026SemaPop: Semantic-Persona Conditioned and Controllable Population Synthesiscited by 0
3 more
  1. 2026Constraint-Aware Counterfactual Editing for Aspect-Based Sentiment Analysiscited by 0
  2. 2026GRAFT: Graph-Distilled Generative Retrieval for Facet-Aware Scientific Literature Explorationcited by 0
  3. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformerscited by 0