# FastFill: Efficient Compatible Model Update

arXiv: https://arxiv.org/abs/2303.04766 · HTML: https://mlsanity.com/paper/2303.04766/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [An Empirical Analysis of Backward Compatibility in Machine Learning Systems](https://mlsanity.com/paper/2008.04572/) (2020)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Unified Representation Learning for Cross Model Compatibility](https://mlsanity.com/paper/2008.04821/) (2020)
- [Asymmetric metric learning for knowledge transfer](https://mlsanity.com/paper/2006.16331/) (2020)
- [Compatibility-aware Heterogeneous Visual Search](https://mlsanity.com/paper/2105.06047/) (2021)
- [Forward Compatible Training for Large-Scale Embedding Retrieval Systems](https://mlsanity.com/paper/2112.02805/) (2021)
- [Learning Backward Compatible Embeddings](https://mlsanity.com/paper/2206.03040/) (2022)
- [Backward-Compatible Prediction Updates: A Probabilistic Approach](https://mlsanity.com/paper/2107.01057/) (2021)
- [Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval](https://mlsanity.com/paper/2204.13919/) (2022)
- [ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training](https://mlsanity.com/paper/2205.06265/) (2022)
- [Distributionally Robust Group Backwards Compatibility](https://mlsanity.com/paper/2112.10290/) (2021)

## Built on by (most cited first, 7 of 7)

- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Global Minimizers of Sigmoid Contrastive Loss](https://mlsanity.com/paper/2509.18552/) (2025)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [Asymmetric Image Retrieval with Cross Model Compatible Ensembles](https://mlsanity.com/paper/2303.17531/) (2023)
- [Prototype Perturbation for Relaxing Alignment Constraints in Backward-Compatible Learning](https://mlsanity.com/paper/2503.14824/) (2025)
- [Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility](https://mlsanity.com/paper/2609.16875/) (2026)
- [Spherical Interpolation for Backward-Compatible Multimodal Representations](https://mlsanity.com/paper/2609.39836/) (2026)

## Abstract (arXiv)

In many retrieval systems the original high dimensional data (e.g., images) is mapped to a lower dimensional feature through a learned embedding model. The task of retrieving the most similar data from a gallery set to a given query data is performed through a similarity comparison on features. When the embedding model is updated, it might produce features that are not comparable/compatible with features already in the gallery computed with the old model. Subsequently, all features in the gallery need to be re-computed using the new embedding model -- a computationally expensive process called backfilling. Recently, compatible representation learning methods have been proposed to avoid backfilling. Despite their relative success, there is an inherent trade-off between the new model performance and its compatibility with the old model. In this work, we introduce FastFill: a compatible model update process using feature alignment and policy based partial backfilling to promptly elevate retrieval performance. We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance of both the training objective and the ordering in online partial backfilling. We propose a new training method for feature alignment between old and new embedding models using uncertainty estimation. Compared to previous works, we obtain significantly improved backfilling results on a variety of datasets: mAP on ImageNet (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%). Further, we demonstrate that when updating a biased model with FastFill, the minority subgroup accuracy gap promptly vanishes with a small fraction of partial backfilling.