# Positive-Congruent Training: Towards Regression-Free Model Updates

arXiv: https://arxiv.org/abs/2011.09161 · HTML: https://mlsanity.com/paper/2011.09161/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 30 of 57)

- [Training Uncertainty-Aware Classifiers with Conformalized Deep Learning](https://mlsanity.com/paper/2205.05878/) (2022)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Monitored Distillation for Positive Congruent Depth Completion](https://mlsanity.com/paper/2203.16034/) (2022)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [MUSCLE: A Model Update Strategy for Compatible LLM Evolution](https://mlsanity.com/paper/2407.09435/) (2024)
- [Backward-Compatible Prediction Updates: A Probabilistic Approach](https://mlsanity.com/paper/2107.01057/) (2021)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Good Teachers Explain: Explanation-Enhanced Knowledge Distillation](https://mlsanity.com/paper/2402.03119/) (2024)
- [Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates](https://mlsanity.com/paper/2105.03048/) (2021)
- [Measuring and Reducing Model Update Regression in Structured Prediction for NLP](https://mlsanity.com/paper/2202.02976/) (2022)
- [AI Model Disgorgement: Methods and Choices](https://mlsanity.com/paper/2304.03545/) (2023)
- [NeuRecover: Regression-Controlled Repair of Deep Neural Networks with Training History](https://mlsanity.com/paper/2203.00191/) (2022)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification](https://mlsanity.com/paper/2203.10622/) (2022)
- [ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training](https://mlsanity.com/paper/2205.06265/) (2022)
- [Backward Compatibility During Data Updates by Weight Interpolation](https://mlsanity.com/paper/2301.10546/) (2023)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)
- [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023)
- [Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems](https://mlsanity.com/paper/2103.11766/) (2021)
- [Darwinian Model Upgrades: Model Evolving with Selective Compatibility](https://mlsanity.com/paper/2210.06954/) (2022)
- [Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AI](https://mlsanity.com/paper/2405.14061/) (2024)
- [Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2405.14715/) (2024)
- [UnCLe: Benchmarking Unsupervised Continual Learning for Depth Completion](https://mlsanity.com/paper/2410.18074/) (2024)
- [Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation](https://mlsanity.com/paper/2604.27405/) (2026)
- [Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates](https://mlsanity.com/paper/2402.17390/) (2024)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [Clinical Trial Active Learning](https://mlsanity.com/paper/2307.11209/) (2023)
- [Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer](https://mlsanity.com/paper/2408.08793/) (2024)
- [Mitigating Negative Flips via Margin Preserving Training](https://mlsanity.com/paper/2511.08322/) (2025)

## Abstract (arXiv)

Reducing inconsistencies in the behavior of different versions of an AI system can be as important in practice as reducing its overall error. In image classification, sample-wise inconsistencies appear as "negative flips": A new model incorrectly predicts the output for a test sample that was correctly classified by the old (reference) model. Positive-congruent (PC) training aims at reducing error rate while at the same time reducing negative flips, thus maximizing congruency with the reference model only on positive predictions, unlike model distillation. We propose a simple approach for PC training, Focal Distillation, which enforces congruence with the reference model by giving more weights to samples that were correctly classified. We also found that, if the reference model itself can be chosen as an ensemble of multiple deep neural networks, negative flips can be further reduced without affecting the new model's accuracy.