mlsanityData to 14 Oct 2026

From Robustness to Improved Generalization and Calibration in Pre-trained Language Models

arXiv 2404.00758 ↗First posted 31 Mar 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Enhancing generalization and uncertainty quantification in pre-trained language models (PLMs) is crucial for their effectiveness and reliability. Building on machine learning research that established the importance of robustness for improving generalization, we investigate the role of representation smoothness, achieved via Jacobian and Hessian regularization, in enhancing PLM performance. Although such regularization methods have proven effective in computer vision, their application in natural language processing (NLP), where PLM inputs are derived from a discrete domain, poses unique challenges. We introduce a novel two-phase regularization approach, JacHess, which minimizes the norms of the Jacobian and Hessian matrices within PLM intermediate representations relative to their inputs. Our evaluation using the GLUE benchmark demonstrates that JacHess significantly improves in-domain generalization and calibration in PLMs, outperforming unregularized fine-tuning and other similar regularization methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

45 of 63 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
33 more
  1. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  4. 2018Adversarially Robust Generalization Requires More Datacited by 318
  5. 2020Calibration of Pre-trained Transformerscited by 315
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  7. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  8. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  9. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  10. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  11. 2017Sobolev Training for Neural Networkscited by 143
  12. 2019Robust Learning with Jacobian Regularizationcited by 129
  13. 2016Robust Large Margin Deep Neural Networkscited by 118
  14. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  15. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  16. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  17. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  18. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  19. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  20. 2015Neural Network with Unbounded Activation Functions is Universal Approximatorcited by 63
  21. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 51
  22. 2020A case for new neural network smoothness constraintscited by 43
  23. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 42
  24. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  25. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  26. 2017Gradient Regularization Improves Accuracy of Discriminative Modelscited by 26
  27. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  28. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  29. 2022Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Taskscited by 13
  30. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  31. 2020Input Hessian Regularization of Neural Networkscited by 7
  32. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  33. 2022Neural Networks as Paths through the Space of Representationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Attribution Patching Lies: Diagnosis and a Second-Order Correctioncited by 3