mlsanityData to 14 Oct 2026

Methods for Estimating and Improving Robustness of Language Models

arXiv 2206.08446 ↗First posted 16 Jun 2022cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions towards enhancing the robustness of LLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
31 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  9. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  10. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  11. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  12. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  13. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  14. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  15. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  16. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 139
  17. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  18. 2019A Primer on PAC-Bayesian Learningcited by 132
  19. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  20. 2018Invariance, Causality and Robustnesscited by 123
  21. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  22. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  23. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  24. 2020In Search of Robust Measures of Generalizationcited by 79
  25. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  26. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  27. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
  28. 2021Confidence Calibration for Domain Generalization under Covariate Shiftcited by 27
  29. 2018Analyzing Compositionality-Sensitivity of NLI Modelscited by 26
  30. 2020Improving QA Generalization by Concurrent Modeling of Multiple Biasescited by 12
  31. 2021Regressive Ensemble for Machine Translation Quality Evaluationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 2.

3 papers.

  1. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  2. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  3. 2025Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable eventscited by 2