# SemEval-2020 Task 5: Counterfactual Recognition

arXiv: https://arxiv.org/abs/2008.00563 · HTML: https://mlsanity.com/paper/2008.00563/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2020.

## Built on by (most cited first, 12 of 12)

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge](https://mlsanity.com/paper/2311.08147/) (2023)
- [I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews](https://mlsanity.com/paper/2104.06893/) (2021)
- [Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysis](https://mlsanity.com/paper/2106.15231/) (2021)
- [The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoning](https://mlsanity.com/paper/2406.19307/) (2024)
- [Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them](https://mlsanity.com/paper/2503.16401/) (2025)
- [Enhancing Language Representation with Constructional Information for Natural Language Understanding](https://mlsanity.com/paper/2306.02819/) (2023)
- [Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models](https://mlsanity.com/paper/2312.11720/) (2023)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [POLygraph: Polish Fake News Dataset](https://mlsanity.com/paper/2407.01393/) (2024)
- [Topic-aware Causal Intervention for Counterfactual Detection](https://mlsanity.com/paper/2409.16668/) (2024)
- [How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset](https://mlsanity.com/paper/2511.06418/) (2025)

## Abstract (arXiv)

We present a counterfactual recognition (CR) task, the shared Task 5 of SemEval-2020. Counterfactuals describe potential outcomes (consequents) produced by actions or circumstances that did not happen or cannot happen and are counter to the facts (antecedent). Counterfactual thinking is an important characteristic of the human cognitive system; it connects antecedents and consequents with causal relations. Our task provides a benchmark for counterfactual recognition in natural language with two subtasks. Subtask-1 aims to determine whether a given sentence is a counterfactual statement or not. Subtask-2 requires the participating systems to extract the antecedent and consequent in a given counterfactual statement. During the SemEval-2020 official evaluation period, we received 27 submissions to Subtask-1 and 11 to Subtask-2. The data, baseline code, and leaderboard can be found at https://competitions.codalab.org/competitions/21691. The data and baseline code are also available at https://zenodo.org/record/3932442.