mlsanityData to 14 Oct 2026

Data Augmentations for Improved (Large) Language Model Generalization

arXiv 2310.12803 ↗First posted 19 Oct 2023cs.LG cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

The reliance of text classifiers on spurious correlations can lead to poor generalization at deployment, raising concerns about their use in safety-critical domains such as healthcare. In this work, we propose to use counterfactual data augmentation, guided by knowledge of the causal structure of the data, to simulate interventions on spurious features and to learn more robust text classifiers. We show that this strategy is appropriate in prediction problems where the label is spuriously correlated with an attribute. Under the assumptions of such problems, we discuss the favorable sample complexity of counterfactual data augmentation, compared to importance re-weighting. Pragmatically, we match examples using auxiliary data, based on diff-in-diff methodology, and use a large language model (LLM) to represent a conditional probability of text. Through extensive experimentation on learning caregiver-invariant predictors of clinical diagnoses from medical narratives and on semi-synthetic data, we demonstrate that our method for simulating interventions improves out-of-distribution (OOD) accuracy compared to baseline invariant learning algorithms.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 49 of its 98 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

49 of 98 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  8. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  11. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  12. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
37 more
  1. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  2. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  3. 2020The Risks of Invariant Risk Minimizationcited by 262
  4. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  5. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  6. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  7. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  8. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 151
  9. 2017Invariant Causal Prediction for Nonlinear Modelscited by 140
  10. 2021On Calibration and Out-of-domain Generalizationcited by 134
  11. 2017Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributionscited by 126
  12. 2019A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelscited by 117
  13. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  14. 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
  15. 2021Model-Based Domain Generalizationcited by 104
  16. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  17. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 65
  18. 2021Causally motivated Shortcut Removal Using Auxiliary Labelscited by 55
  19. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 55
  20. 2019Enhancing Clinical Concept Extraction with Contextual Embeddingscited by 48
  21. 2020Generative Interventions for Causal Learningcited by 45
  22. 2021An Investigation of the (In)effectiveness of Counterfactually Augmented Datacited by 42
  23. 2021Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlationscited by 41
  24. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 37
  25. 2022Invariant and Transportable Representations for Anti-Causal Domain Shiftscited by 37
  26. 2023Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropycited by 30
  27. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  28. 2021Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fixcited by 27
  29. 2020Invariant Representation Learning for Treatment Effect Estimationcited by 25
  30. 2023Out-of-Domain Robustness via Targeted Augmentationscited by 24
  31. 2022Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentationcited by 18
  32. 2019A Unifying Causal Framework for Analyzing Dataset Shift-stable Learning Algorithmscited by 17
  33. 2021Optimization-based Causal Estimation from Heterogenous Environmentscited by 16
  34. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15
  35. 2018Clinical Concept Extraction with Contextual Word Embeddingcited by 12
  36. 2022Malign Overfitting: Interpolation Can Provably Preclude Invariancecited by 6
  37. 2022In the Eye of the Beholder: Robust Prediction with Causal User Modelingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  2. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 7
  3. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  4. 2024Estimating Causal Effects of Text Interventions Leveraging LLMscited by 4
  5. 2025Learning from the Undesirable: Robust Adaptation of Language Models without Forgettingcited by 3
  6. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2
  7. 2025An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentationcited by 2
  8. 2026Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Modelscited by 1
  9. 2026SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scalingcited by 1
  10. 2024FaceMixup: Enhancing Facial Expression Recognition through Mixed Face Regularizationcited by 0
  11. 2024Causal Fine-Tuning under Latent Confounded Shiftcited by 0
  12. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
2 more
  1. 2025Expert-guided Clinical Text Augmentation via Query-Based Model Collaborationcited by 0
  2. 2026Counterfactual Self-Evolving Agents for Evidence-Grounded Reasoningcited by 0