mlsanityData to 7 Oct 2026

Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates

arXiv 2105.03048 ↗First posted 7 May 2021cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Behavior of deep neural networks can be inconsistent between different versions. Regressions during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regression measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimization problem, and further reduce it into a relaxed form which can be approximately optimized through knowledge distillation training method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CheckList behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

27 of 46 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
15 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  7. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  8. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  9. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  10. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  11. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  12. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  13. 2020Towards Backward-Compatible Representation Learningcited by 74
  14. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  15. 2020Lifelong Language Knowledge Distillationcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 5; 2025: 1; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Predictive Churn with the Set of Good Modelscited by 15
  2. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  3. 2022Measuring and Reducing Model Update Regression in Structured Prediction for NLPcited by 11
  4. 2024LIRE: listwise reward enhancement for preference alignmentcited by 11
  5. 2022ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Trainingcited by 7
  6. 2023Backward Compatibility During Data Updates by Weight Interpolationcited by 7
  7. 2022Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrievalcited by 6
  8. 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
  9. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3
  10. 2024FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimizationcited by 2
  11. 2024Evaluating the Robustness of Adverse Drug Event Classification Models Using Templatescited by 1
  12. 2026No Universal Signal Predicts Sample-Level LLM Regression under Version Updatescited by 0
1 more
  1. 2026Faster Block-Diffusion Serving with Distribution-Free Risk Guaranteescited by 0