# Towards Backward-Compatible Representation Learning

arXiv: https://arxiv.org/abs/2003.11942 · HTML: https://mlsanity.com/paper/2003.11942/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2020.

## Built on by (most cited first, 30 of 75)

- [Forward Compatible Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2203.06953/) (2022)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [Compatibility-aware Heterogeneous Visual Search](https://mlsanity.com/paper/2105.06047/) (2021)
- [Towards Universal Backward-Compatible Representation Learning](https://mlsanity.com/paper/2203.01583/) (2022)
- [Forward Compatible Training for Large-Scale Embedding Retrieval Systems](https://mlsanity.com/paper/2112.02805/) (2021)
- [Improving Federated Learning Face Recognition via Privacy-Agnostic Clusters](https://mlsanity.com/paper/2201.12467/) (2022)
- [Continual Learning for Visual Search with Backward Consistent Feature Embedding](https://mlsanity.com/paper/2205.13384/) (2022)
- [Is Self-Supervised Learning More Robust Than Supervised Learning?](https://mlsanity.com/paper/2206.05259/) (2022)
- [Learning Backward Compatible Embeddings](https://mlsanity.com/paper/2206.03040/) (2022)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [MUSCLE: A Model Update Strategy for Compatible LLM Evolution](https://mlsanity.com/paper/2407.09435/) (2024)
- [Backward-Compatible Prediction Updates: A Probabilistic Approach](https://mlsanity.com/paper/2107.01057/) (2021)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates](https://mlsanity.com/paper/2105.03048/) (2021)
- [Measuring and Reducing Model Update Regression in Structured Prediction for NLP](https://mlsanity.com/paper/2202.02976/) (2022)
- [Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval](https://mlsanity.com/paper/2305.02610/) (2023)
- [Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning](https://mlsanity.com/paper/2108.02959/) (2021)
- [Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval](https://mlsanity.com/paper/2204.13919/) (2022)
- [AI Model Disgorgement: Methods and Choices](https://mlsanity.com/paper/2304.03545/) (2023)
- [Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation](https://mlsanity.com/paper/2108.03372/) (2021)
- [It's All in the Head: Representation Knowledge Distillation through Classifier Sharing](https://mlsanity.com/paper/2201.06945/) (2022)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [AutoML in The Wild: Obstacles, Workarounds, and Expectations](https://mlsanity.com/paper/2302.10827/) (2023)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training](https://mlsanity.com/paper/2205.06265/) (2022)
- [Backward Compatibility During Data Updates by Weight Interpolation](https://mlsanity.com/paper/2301.10546/) (2023)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)

## Abstract (arXiv)

We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions and are learned via different neural network architectures and loss functions. Compatible means that, if such features are used to compare images, then "new" features can be compared directly to "old" features, so they can be used interchangeably. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models, a process known as backfilling. Backward compatibility is critical to quickly deploy new embedding models that leverage ever-growing large-scale training datasets and improvements in deep learning architectures and training methods. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning. In experiments on learning embeddings for face recognition, models trained with BCT successfully achieve backward compatibility without sacrificing accuracy, thus enabling backfill-free model updates of visual embeddings.