# Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems

arXiv: https://arxiv.org/abs/2108.05053 · HTML: https://mlsanity.com/paper/2108.05053/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [Ludwig: a type-based declarative deep learning toolbox](https://mlsanity.com/paper/1909.07930/) (2019)
- [Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation](https://mlsanity.com/paper/2010.10363/) (2020)
- [Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices](https://mlsanity.com/paper/1909.06349/) (2019)
- [Overton: A Data System for Monitoring and Improving Machine-Learned Products](https://mlsanity.com/paper/1909.05372/) (2019)
- [On the Downstream Performance of Compressed Word Embeddings](https://mlsanity.com/paper/1909.01264/) (2019)
- [Understanding the Downstream Instability of Word Embeddings](https://mlsanity.com/paper/2003.04983/) (2020)
- [ARDA: Automatic Relational Data Augmentation for Machine Learning](https://mlsanity.com/paper/2003.09758/) (2020)

## Built on by (most cited first, 6 of 6)

- [Looper: An end-to-end ML platform for product decisions](https://mlsanity.com/paper/2110.07554/) (2021)
- [Fix your Models by Fixing your Datasets](https://mlsanity.com/paper/2112.07844/) (2021)
- [SHiFT: An Efficient, Flexible Search Engine for Transfer Learning](https://mlsanity.com/paper/2204.01457/) (2022)
- [Efficient Multi-stage Inference on Tabular Data](https://mlsanity.com/paper/2303.11580/) (2023)
- [Managed Geo-Distributed Feature Store: Architecture and System Design](https://mlsanity.com/paper/2305.20077/) (2023)
- [Learning Source Acquisition Policies by Offline Planning](https://mlsanity.com/paper/2609.14299/) (2026)

## Abstract (arXiv)

The industrial machine learning pipeline requires iterating on model features, training and deploying models, and monitoring deployed models at scale. Feature stores were developed to manage and standardize the engineer's workflow in this end-to-end pipeline, focusing on traditional tabular feature data. In recent years, however, model development has shifted towards using self-supervised pretrained embeddings as model features. Managing these embeddings and the downstream systems that use them introduces new challenges with respect to managing embedding training data, measuring embedding quality, and monitoring downstream models that use embeddings. These challenges are largely unaddressed in standard feature stores. Our goal in this tutorial is to introduce the feature store system and discuss the challenges and current solutions to managing these new embedding-centric pipelines.