# Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements

arXiv: https://arxiv.org/abs/2405.02581 · HTML: https://mlsanity.com/paper/2405.02581/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 7 of 7)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [Maximally Compact and Separated Features with Regular Polytope Networks](https://mlsanity.com/paper/2301.06116/) (2023)

## Built on by (most cited first, 8 of 8)

- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models](https://mlsanity.com/paper/2506.00037/) (2025)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer](https://mlsanity.com/paper/2408.08793/) (2024)
- [Prototype Perturbation for Relaxing Alignment Constraints in Backward-Compatible Learning](https://mlsanity.com/paper/2503.14824/) (2025)
- [A Stationary (and Therefore Compatible) Representation is All You Need](https://mlsanity.com/paper/2606.12488/) (2026)
- [RST: certifying and constructing prescribed information in variational autoencoders](https://mlsanity.com/paper/2605.18224/) (2026)
- [Spherical Interpolation for Backward-Compatible Multimodal Representations](https://mlsanity.com/paper/2609.39836/) (2026)

## Abstract (arXiv)

Learning compatible representations enables the interchangeable use of semantic features as models are updated over time. This is particularly relevant in search and retrieval systems where it is crucial to avoid reprocessing of the gallery images with the updated model. While recent research has shown promising empirical evidence, there is still a lack of comprehensive theoretical understanding about learning compatible representations. In this paper, we demonstrate that the stationary representations learned by the d-Simplex fixed classifier optimally approximate compatibility representation according to the two inequality constraints of its formal definition. This not only establishes a solid foundation for future works in this line of research but also presents implications that can be exploited in practical learning scenarios. An exemplary application is the now-standard practice of downloading and fine-tuning new pre-trained models. Specifically, we show the strengths and critical issues of stationary representations in the case in which a model undergoing sequential fine-tuning is asynchronously replaced by downloading a better-performing model pre-trained elsewhere. Such a representation enables seamless delivery of retrieval service (i.e., no reprocessing of gallery images) and offers improved performance without operational disruptions during model replacement. Code available at: https://github.com/miccunifi/iamcl2r.