mlsanityData to 7 Oct 2026

Distributionally Robust Language Modeling

arXiv 1909.02060 ↗First posted 4 Sep 2019cs.CL cs.LG stat.MLcited by 142

Not on the map. Why?

Abstract (arXiv)

Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., restaurant reviews). In this paper, we first show that training on text outside the test distribution can degrade test performance when using standard maximum likelihood (MLE) training. To remedy this without the knowledge of the test distribution, we propose an approach which trains a model that performs well over a wide range of potential test distributions. In particular, we derive a new distributionally robust optimization (DRO) procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution. Our approach, called topic conditional value at risk (topic CVaR), obtains a 5.5 point perplexity reduction over MLE when the language models are trained on a mixture of Yelp reviews and news and tested only on reviews.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 30; 2023: 23; 2024: 24; 2025: 24; 2026: 11.

142 papers.

Sort:

By area:

  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  4. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  5. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  6. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  7. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  8. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  9. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  10. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  11. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  12. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
38 more
  1. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  2. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  3. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  4. 2021Examining and Combating Spurious Features under Distribution Shiftcited by 64
  5. 2021Editing a classifier by rewriting its prediction rulescited by 64
  6. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  7. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  8. 2021DORO: Distributional and Outlier Robust Optimizationcited by 54
  9. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  10. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  11. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  12. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  13. 2021Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3D deep generative modelscited by 36
  14. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  15. 2022Minimax Regret Optimization for Robust Machine Learning under Distribution Shiftcited by 31
  16. 2022Learning to Solve Routing Problems via Distributionally Robust Optimizationcited by 31
  17. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29
  18. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  19. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  20. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  21. 2022What a Creole Wants, What a Creole Needscited by 27
  22. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  23. 2023Re-weighting Based Group Fairness Regularization via Classwise Robust Optimizationcited by 20
  24. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
  25. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  26. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  27. 2021Distributionally Robust Multilingual Machine Translationcited by 18
  28. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  29. 2021On Language Models for Creolescited by 17
  30. 2022Distributionally Robust Models with Parametric Likelihood Ratioscited by 17
  31. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
  32. 2021Predict then Interpolate: A Simple Algorithm to Learn Stable Classifierscited by 16
  33. 2022Example-based Hypernetworks for Out-of-Distribution Generalizationcited by 16
  34. 2022Learning to Split for Automatic Bias Detectioncited by 16
  35. 2022Near-Optimal Algorithms for Group Distributionally Robust Optimization and Beyondcited by 16
  36. 2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
  37. 2021Invariant Language Modelingcited by 15
  38. 2021Unified Group Fairness on Federated Learningcited by 15

50 most cited of 142 All 142 (JSON)