# Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP

arXiv: https://arxiv.org/abs/2110.03618 · HTML: https://mlsanity.com/paper/2110.03618/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 87 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [The Description Length of Deep Learning Models](https://mlsanity.com/paper/1802.07044/) (2018)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates](https://mlsanity.com/paper/2005.00649/) (2020)
- [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://mlsanity.com/paper/1908.05204/) (2019)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [Translationese as a Language in "Multilingual" NMT](https://mlsanity.com/paper/1911.03823/) (2019)
- [Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation](https://mlsanity.com/paper/1911.03362/) (2019)
- [Semi-Supervised Learning, Causality and the Conditional Cluster Assumption](https://mlsanity.com/paper/1905.12081/) (2019)
- [Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Features](https://mlsanity.com/paper/1807.07879/) (2018)
- [The Source-Target Domain Mismatch Problem in Machine Translation](https://mlsanity.com/paper/1909.13151/) (2019)
- [Predicting Directionality in Causal Relations in Text](https://mlsanity.com/paper/2103.13606/) (2021)
- [Predicting the Effectiveness of Self-Training: Application to Sentiment Classification](https://mlsanity.com/paper/1601.03288/) (2016)

## Built on by (most cited first, 21 of 21)

- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models](https://mlsanity.com/paper/2210.12023/) (2022)
- [Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions](https://mlsanity.com/paper/2207.14251/) (2022)
- [From Statistical to Causal Learning](https://mlsanity.com/paper/2204.00607/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Language](https://mlsanity.com/paper/2204.04487/) (2022)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [Sequential Learning Of Neural Networks for Prequential MDL](https://mlsanity.com/paper/2210.07931/) (2022)
- [Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance](https://mlsanity.com/paper/2205.02293/) (2022)
- [Counterfactually Augmented Data and Unintended Bias: The Case of Sexism and Hate Speech Detection](https://mlsanity.com/paper/2205.04238/) (2022)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10381/) (2022)
- [Tackling Concept Shift in Text Classification using Entailment-style Modeling](https://mlsanity.com/paper/2311.03320/) (2023)
- [Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemaking](https://mlsanity.com/paper/2601.11459/) (2026)
- [Fair Text Classification via Transferable Representations](https://mlsanity.com/paper/2503.07691/) (2025)
- [Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulation](https://mlsanity.com/paper/2308.01184/) (2023)
- [Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification](https://mlsanity.com/paper/2410.00179/) (2024)
- [Causal vs. Anticausal merging of predictors](https://mlsanity.com/paper/2501.08426/) (2025)
- [Psychologically-Inspired Causal Prompts](https://mlsanity.com/paper/2305.01764/) (2023)
- [On the Role of Priors in Bayesian Causal Learning](https://mlsanity.com/paper/2504.01424/) (2025)

## Abstract (arXiv)

The principle of independent causal mechanisms (ICM) states that generative processes of real world data consist of independent modules which do not influence or inform each other. While this idea has led to fruitful developments in the field of causal inference, it is not widely-known in the NLP community. In this work, we argue that the causal direction of the data collection process bears nontrivial implications that can explain a number of published NLP findings, such as differences in semi-supervised learning (SSL) and domain adaptation (DA) performance across different settings. We categorize common NLP tasks according to their causal direction and empirically assay the validity of the ICM principle for text data using minimum description length. We conduct an extensive meta-analysis of over 100 published SSL and 30 DA studies, and find that the results are consistent with our expectations based on causal insights. This work presents the first attempt to analyze the ICM principle in NLP, and provides constructive suggestions for future modeling choices. Code available at https://github.com/zhijing-jin/icm4nlp