# Improving Commonsense Causal Reasoning by Adversarial Training and Data Augmentation

arXiv: https://arxiv.org/abs/2101.04966 · HTML: https://mlsanity.com/paper/2101.04966/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [When Choosing Plausible Alternatives, Clever Hans can be Clever](https://mlsanity.com/paper/1911.00225/) (2019)
- [Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning](https://mlsanity.com/paper/2004.14074/) (2020)
- [OpenHowNet: An Open Sememe-based Lexical Knowledge Base](https://mlsanity.com/paper/1901.09957/) (2019)
- [Learning to Rank for Plausible Plausibility](https://mlsanity.com/paper/1906.02079/) (2019)
- [Bridging the Knowledge Gap: Enhancing Question Answering with World and Domain Knowledge](https://mlsanity.com/paper/1910.07429/) (2019)
- [Learning Fine-Grained Knowledge about Contingent Relations between Everyday Events](https://mlsanity.com/paper/1708.09450/) (2017)

## Built on by (most cited first, 9 of 9)

- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective](https://mlsanity.com/paper/2305.05191/) (2023)
- [ROCK: Causal Inference Principles for Reasoning about Commonsense Causality](https://mlsanity.com/paper/2202.00436/) (2022)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://mlsanity.com/paper/2205.08794/) (2022)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Modeling Associative Reasoning Processes](https://mlsanity.com/paper/2201.00716/) (2022)
- [Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration](https://mlsanity.com/paper/2509.21530/) (2025)

## Abstract (arXiv)

Determining the plausibility of causal relations between clauses is a commonsense reasoning task that requires complex inference ability. The general approach to this task is to train a large pretrained language model on a specific dataset. However, the available training data for the task is often scarce, which leads to instability of model training or reliance on the shallow features of the dataset. This paper presents a number of techniques for making models more robust in the domain of causal reasoning. Firstly, we perform adversarial training by generating perturbed inputs through synonym substitution. Secondly, based on a linguistic theory of discourse connectives, we perform data augmentation using a discourse parser for detecting causally linked clauses in large text, and a generative language model for generating distractors. Both methods boost model performance on the Choice of Plausible Alternatives (COPA) dataset, as well as on a Balanced COPA dataset, which is a modified version of the original data that has been developed to avoid superficial cues, leading to a more challenging benchmark. We show a statistically significant improvement in performance and robustness on both datasets, even with only a small number of additionally generated data points.