# Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer

arXiv: https://arxiv.org/abs/2408.08793 · HTML: https://mlsanity.com/paper/2408.08793/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Unified Representation Learning for Cross Model Compatibility](https://mlsanity.com/paper/2008.04821/) (2020)
- [Asymmetric metric learning for knowledge transfer](https://mlsanity.com/paper/2006.16331/) (2020)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [Compatibility-aware Heterogeneous Visual Search](https://mlsanity.com/paper/2105.06047/) (2021)
- [Towards Universal Backward-Compatible Representation Learning](https://mlsanity.com/paper/2203.01583/) (2022)
- [Forward Compatible Training for Large-Scale Embedding Retrieval Systems](https://mlsanity.com/paper/2112.02805/) (2021)
- [Continual Learning for Visual Search with Backward Consistent Feature Embedding](https://mlsanity.com/paper/2205.13384/) (2022)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements](https://mlsanity.com/paper/2405.02581/) (2024)
- [CL2R: Compatible Lifelong Learning Representations](https://mlsanity.com/paper/2211.09032/) (2022)
- [Metric Compatible Training for Online Backfilling in Large-Scale Retrieval](https://mlsanity.com/paper/2301.03767/) (2023)

## Built on by (most cited first, 3 of 3)

- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Mitigating Negative Flips via Margin Preserving Training](https://mlsanity.com/paper/2511.08322/) (2025)
- [Spherical Interpolation for Backward-Compatible Multimodal Representations](https://mlsanity.com/paper/2609.39836/) (2026)

## Abstract (arXiv)

Visual retrieval systems face significant challenges when updating models with improved representations due to misalignment between the old and new representations. The costly and resource-intensive backfilling process involves recalculating feature vectors for images in the gallery set whenever a new model is introduced. To address this, prior research has explored backward-compatible training methods that enable direct comparisons between new and old representations without backfilling. Despite these advancements, achieving a balance between backward compatibility and the performance of independently trained models remains an open problem. In this paper, we address it by expanding the representation space with additional dimensions and learning an orthogonal transformation to achieve compatibility with old models and, at the same time, integrate new information. This transformation preserves the original feature space's geometry, ensuring that our model aligns with previous versions while also learning new data. Our Orthogonal Compatible Aligned (OCA) approach eliminates the need for re-indexing during model updates and ensures that features can be compared directly across different model updates without additional mapping functions. Experimental results on CIFAR-100 and ImageNet-1k demonstrate that our method not only maintains compatibility with previous models but also achieves state-of-the-art accuracy, outperforming several existing methods.