# Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems

arXiv: https://arxiv.org/abs/2103.11766 · HTML: https://mlsanity.com/paper/2103.11766/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Fairness Under Composition](https://mlsanity.com/paper/1806.06122/) (2018)
- [An Empirical Analysis of Backward Compatibility in Machine Learning Systems](https://mlsanity.com/paper/2008.04572/) (2020)
- [Unified Representation Learning for Cross Model Compatibility](https://mlsanity.com/paper/2008.04821/) (2020)
- [On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems](https://mlsanity.com/paper/1611.08309/) (2016)

## Built on by (most cited first, 5 of 5)

- [TaskMet: Task-Driven Metric Learning for Model Learning](https://mlsanity.com/paper/2312.05250/) (2023)
- [OneLog: Towards End-to-End Training in Software Log Anomaly Detection](https://mlsanity.com/paper/2104.07324/) (2021)
- [Behavior quantification as the missing link between fields: Tools for digital psychiatry and their role in the future of neurobiology](https://mlsanity.com/paper/2305.15385/) (2023)
- [Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems](https://mlsanity.com/paper/2301.03837/) (2023)
- [On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval](https://mlsanity.com/paper/2506.11499/) (2025)

## Abstract (arXiv)

Machine-learning systems such as self-driving cars or virtual assistants are composed of a large number of machine-learning models that recognize image content, transcribe speech, analyze natural language, infer preferences, rank options, etc. Models in these systems are often developed and trained independently, which raises an obvious concern: Can improving a machine-learning model make the overall system worse? We answer this question affirmatively by showing that improving a model can deteriorate the performance of downstream models, even after those downstream models are retrained. Such self-defeating improvements are the result of entanglement between the models in the system. We perform an error decomposition of systems with multiple machine-learning models, which sheds light on the types of errors that can lead to self-defeating improvements. We also present the results of experiments which show that self-defeating improvements emerge in a realistic stereo-based detection system for cars and pedestrians.