mlsanityData to 14 Oct 2026

Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP

arXiv 2110.03618 ↗First posted 7 Oct 2021cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

The principle of independent causal mechanisms (ICM) states that generative processes of real world data consist of independent modules which do not influence or inform each other. While this idea has led to fruitful developments in the field of causal inference, it is not widely-known in the NLP community. In this work, we argue that the causal direction of the data collection process bears nontrivial implications that can explain a number of published NLP findings, such as differences in semi-supervised learning (SSL) and domain adaptation (DA) performance across different settings. We categorize common NLP tasks according to their causal direction and empirically assay the validity of the ICM principle for text data using minimum description length. We conduct an extensive meta-analysis of over 100 published SSL and 30 DA studies, and find that the results are consistent with our expectations based on causal insights. This work presents the first attempt to analyze the ICM principle in NLP, and provides constructive suggestions for future modeling choices. Code available at https://github.com/zhijing-jin/icm4nlp

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 26 of its 87 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

26 of 87 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  6. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  7. 2018Marian: Fast Neural Machine Translation in C++cited by 203
  8. 2015Semi-supervised Sequence Learningcited by 198
  9. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  10. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  11. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  12. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
14 more
  1. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  2. 2018The Description Length of Deep Learning Modelscited by 72
  3. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  4. 2018How to Make Causal Inferences Using Textscited by 59
  5. 2020Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimatescited by 59
  6. 2019On The Evaluation of Machine Translation Systems Trained With Back-Translationcited by 40
  7. 2019Translationese in Machine Translation Evaluationcited by 38
  8. 2019Translationese as a Language in "Multilingual" NMTcited by 36
  9. 2019Domain, Translationese and Noise in Synthetic Data for Neural Machine Translationcited by 26
  10. 2019Semi-Supervised Learning, Causality and the Conditional Cluster Assumptioncited by 15
  11. 2018Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Featurescited by 8
  12. 2019The Source-Target Domain Mismatch Problem in Machine Translationcited by 8
  13. 2021Predicting Directionality in Causal Relations in Textcited by 8
  14. 2016Predicting the Effectiveness of Self-Training: Application to Sentiment Classificationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 6; 2024: 2; 2025: 4; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 70
  2. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  3. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  4. 2022From Statistical to Causal Learningcited by 40
  5. 2023Causal Reinforcement Learning: A Surveycited by 23
  6. 2024Cause and Effect: Can Large Language Models Truly Understand Causality?cited by 23
  7. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  8. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
  9. 2022Sequential Learning Of Neural Networks for Prequential MDLcited by 13
  10. 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performancecited by 11
  11. 2022Counterfactually Augmented Data and Unintended Bias: The Case of Sexism and Hate Speech Detectioncited by 11
  12. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
9 more
  1. 2022To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answeringcited by 5
  2. 2023Tackling Concept Shift in Text Classification using Entailment-style Modelingcited by 4
  3. 2026Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemakingcited by 4
  4. 2025Fair Text Classification via Transferable Representationscited by 3
  5. 2023Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulationcited by 1
  6. 2024Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classificationcited by 1
  7. 2025Causal vs. Anticausal merging of predictorscited by 1
  8. 2023Psychologically-Inspired Causal Promptscited by 0
  9. 2025On the Role of Priors in Bayesian Causal Learningcited by 0