mlsanityData to 7 Oct 2026

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

arXiv 2302.02080 ↗First posted 4 Feb 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression errors. This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance. Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backward-compatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regression errors by 62% on average, outperforming the strongest baseline by an average of 25%.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

40 of 52 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
28 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  2. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2015Pointer Networkscited by 882
  5. 2016Pixel Recurrent Neural Networkscited by 814
  6. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  7. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  8. 2016Language Modeling with Gated Convolutional Networkscited by 688
  9. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 217
  10. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  11. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  12. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  13. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 125
  14. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  15. 2020Towards Backward-Compatible Representation Learningcited by 74
  16. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  17. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  18. 2021Learning Compatible Embeddingscited by 37
  19. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  20. 2020Lifelong Language Knowledge Distillationcited by 31
  21. 2020Unified Representation Learning for Cross Model Compatibilitycited by 30
  22. 2021Compatibility-aware Heterogeneous Visual Searchcited by 22
  23. 2021Backward-Compatible Prediction Updates: A Probabilistic Approachcited by 14
  24. 2021Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updatescited by 13
  25. 2022Measuring and Reducing Model Update Regression in Structured Prediction for NLPcited by 11
  26. 2018Towards Visual Feature Translationcited by 8
  27. 2023Backward Compatibility During Data Updates by Weight Interpolationcited by 7
  28. 2019Goal-Embedded Dual Hierarchical Model for Task-Oriented Dialogue Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 1.

6 papers.

  1. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  2. 2023Can Model Fusing Help Transformers in Long Document Classification? An Empirical Studycited by 3
  3. 2024FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimizationcited by 2
  4. 2025Regression-aware Continual Learning for Android Malware Detectioncited by 2
  5. 2025SafeFix: Targeted Model Repair via Controlled Image Generationcited by 0
  6. 2026No Universal Signal Predicts Sample-Level LLM Regression under Version Updatescited by 0