# Learning Compatible Embeddings

arXiv: https://arxiv.org/abs/2108.01958 · HTML: https://mlsanity.com/paper/2108.01958/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [Deep Image Retrieval: Learning global representations for image search](https://mlsanity.com/paper/1604.01325/) (2016)
- [L2-constrained Softmax Loss for Discriminative Face Verification](https://mlsanity.com/paper/1703.09507/) (2017)
- [Features for Multi-Target Multi-Camera Tracking and Re-Identification](https://mlsanity.com/paper/1803.10859/) (2018)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Appearance-Preserving 3D Convolution for Video-based Person Re-identification](https://mlsanity.com/paper/2007.08434/) (2020)
- [DeepID3: Face Recognition with Very Deep Neural Networks](https://mlsanity.com/paper/1502.00873/) (2015)
- [DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer](https://mlsanity.com/paper/1707.01220/) (2017)

## Built on by (most cited first, 30 of 37)

- [Forward Compatible Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2203.06953/) (2022)
- [Towards Universal Backward-Compatible Representation Learning](https://mlsanity.com/paper/2203.01583/) (2022)
- [Forward Compatible Training for Large-Scale Embedding Retrieval Systems](https://mlsanity.com/paper/2112.02805/) (2021)
- [Improving Federated Learning Face Recognition via Privacy-Agnostic Clusters](https://mlsanity.com/paper/2201.12467/) (2022)
- [Learning Backward Compatible Embeddings](https://mlsanity.com/paper/2206.03040/) (2022)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval](https://mlsanity.com/paper/2305.02610/) (2023)
- [Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval](https://mlsanity.com/paper/2204.13919/) (2022)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompt](https://mlsanity.com/paper/2310.13024/) (2023)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework and A New Benchmark](https://mlsanity.com/paper/2210.09835/) (2022)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)
- [CL2R: Compatible Lifelong Learning Representations](https://mlsanity.com/paper/2211.09032/) (2022)
- [Metric Compatible Training for Online Backfilling in Large-Scale Retrieval](https://mlsanity.com/paper/2301.03767/) (2023)
- [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023)
- [Binary Embedding-based Retrieval at Tencent](https://mlsanity.com/paper/2302.08714/) (2023)
- [Darwinian Model Upgrades: Model Evolving with Selective Compatibility](https://mlsanity.com/paper/2210.06954/) (2022)
- [Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2405.14715/) (2024)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [Dynamic Gradient Reactivation for Backward Compatible Person Re-identification](https://mlsanity.com/paper/2207.05658/) (2022)
- [Learning Repeatable Speech Embeddings Using An Intra-class Correlation Regularizer](https://mlsanity.com/paper/2310.17049/) (2023)
- [Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer](https://mlsanity.com/paper/2408.08793/) (2024)
- [Towards Privacy-Preserving, Real-Time and Lossless Feature Matching](https://mlsanity.com/paper/2208.00214/) (2022)
- [Asymmetric Image Retrieval with Cross Model Compatible Ensembles](https://mlsanity.com/paper/2303.17531/) (2023)
- [MixBCT: Towards Self-Adapting Backward-Compatible Training](https://mlsanity.com/paper/2308.06948/) (2023)
- [Lifelong Person Re-Identification with Backward-Compatibility](https://mlsanity.com/paper/2403.10022/) (2024)
- [When Embedding Models Meet: Procrustes Bounds and Applications](https://mlsanity.com/paper/2510.13406/) (2025)
- [Towards Test-time Efficient Visual Place Recognition via Asymmetric Query Processing](https://mlsanity.com/paper/2512.13055/) (2025)

## Abstract (arXiv)

Achieving backward compatibility when rolling out new models can highly reduce costs or even bypass feature re-encoding of existing gallery images for in-production visual retrieval systems. Previous related works usually leverage losses used in knowledge distillation which can cause performance degradations or not guarantee compatibility. To address these issues, we propose a general framework called Learning Compatible Embeddings (LCE) which is applicable for both cross model compatibility and compatible training in direct/forward/backward manners. Our compatibility is achieved by aligning class centers between models directly or via a transformation, and restricting more compact intra-class distributions for the new model. Experiments are conducted in extensive scenarios such as changes of training dataset, loss functions, network architectures as well as feature dimensions, and demonstrate that LCE efficiently enables model compatibility with marginal sacrifices of accuracies. The code will be available at https://github.com/IrvingMeng/LCE.