# Learning Backward Compatible Embeddings

arXiv: https://arxiv.org/abs/2206.03040 · HTML: https://mlsanity.com/paper/2206.03040/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [AliGraph: A Comprehensive Graph Neural Network Platform](https://mlsanity.com/paper/1902.08730/) (2019)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [DynGEM: Deep Embedding Method for Dynamic Graphs](https://mlsanity.com/paper/1805.11273/) (2018)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://mlsanity.com/paper/1902.00508/) (2019)
- [Dynamic Word Embeddings for Evolving Semantic Discovery](https://mlsanity.com/paper/1703.00607/) (2017)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Are Girls Neko or Sh=ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization](https://mlsanity.com/paper/1906.01622/) (2019)

## Built on by (most cited first, 16 of 16)

- [PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning](https://mlsanity.com/paper/2404.00776/) (2024)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [AutoML in The Wild: Obstacles, Workarounds, and Expectations](https://mlsanity.com/paper/2302.10827/) (2023)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)
- [Binary Embedding-based Retrieval at Tencent](https://mlsanity.com/paper/2302.08714/) (2023)
- [Darwinian Model Upgrades: Model Evolving with Selective Compatibility](https://mlsanity.com/paper/2210.06954/) (2022)
- [Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2405.14715/) (2024)
- [Large-to-small Image Resolution Asymmetry in Deep Metric Learning](https://mlsanity.com/paper/2210.05463/) (2022)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [MixBCT: Towards Self-Adapting Backward-Compatible Training](https://mlsanity.com/paper/2308.06948/) (2023)
- [A Scalable and Efficient Signal Integration System for Job Matching](https://mlsanity.com/paper/2507.09797/) (2025)
- [When Embedding Models Meet: Procrustes Bounds and Applications](https://mlsanity.com/paper/2510.13406/) (2025)
- [BioSequence2Vec: Efficient Embedding Generation For Biological Sequences](https://mlsanity.com/paper/2304.00291/) (2023)
- [Vector Linking via Cross-Model Local Isometric Consistency](https://mlsanity.com/paper/2605.31100/) (2026)
- [RAFEN -- Regularized Alignment Framework for Embeddings of Nodes](https://mlsanity.com/paper/2303.01926/) (2023)
- [Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility](https://mlsanity.com/paper/2609.16875/) (2026)

## Abstract (arXiv)

Embeddings, low-dimensional vector representation of objects, are fundamental in building modern machine learning systems. In industrial settings, there is usually an embedding team that trains an embedding model to solve intended tasks (e.g., product recommendation). The produced embeddings are then widely consumed by consumer teams to solve their unintended tasks (e.g., fraud detection). However, as the embedding model gets updated and retrained to improve performance on the intended task, the newly-generated embeddings are no longer compatible with the existing consumer models. This means that historical versions of the embeddings can never be retired or all consumer teams have to retrain their models to make them compatible with the latest version of the embeddings, both of which are extremely costly in practice. Here we study the problem of embedding version updates and their backward compatibility. We formalize the problem where the goal is for the embedding team to keep updating the embedding version, while the consumer teams do not have to retrain their models. We develop a solution based on learning backward compatible embeddings, which allows the embedding model version to be updated frequently, while also allowing the latest version of the embedding to be quickly transformed into any backward compatible historical version of it, so that consumer teams do not have to retrain their models. Under our framework, we explore six methods and systematically evaluate them on a real-world recommender system application. We show that the best method, which we call BC-Aligner, maintains backward compatibility with existing unintended tasks even after multiple model version updates. Simultaneously, BC-Aligner achieves the intended task performance similar to the embedding model that is solely optimized for the intended task.