# Memory Layer: Train the In-Model Cache for Recommendation Models

arXiv: https://arxiv.org/abs/2607.25110 · HTML: https://mlsanity.com/paper/2607.25110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 11 of 11)

- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [IntTower: the Next Generation of Two-Tower Model for Pre-Ranking System](https://mlsanity.com/paper/2210.09890/) (2022)
- [RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure](https://mlsanity.com/paper/2211.05239/) (2022)
- [SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs](https://mlsanity.com/paper/2511.14881/) (2025)
- [Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.14961/) (2024)
- [RankTower: A Synergistic Framework for Enhancing Two-Tower Pre-Ranking Model](https://mlsanity.com/paper/2407.12385/) (2024)
- [Target-Aware Early Stage Ranking](https://mlsanity.com/paper/2511.21095/) (2025)

## Abstract (arXiv)

Early ranking stages in recommendation systems precompute item embeddings and cache them in-model for scoring within strict latency constraints. Because this cache exists only at serving time, outside the training loop, training and serving use different item representations, a structural discrepancy that limits quality and adds operational fragility. We show that co-designing the training and serving paths removes this representation discrepancy at its source. We introduce the memory layer, an in-model key-value embedding cache co-trained with the model: the item tower writes embeddings during training and the model reads them at serving, one source of truth for item representations by construction. Always-on embeddings cover items not yet cached, so every item receives a prediction, and the design consolidates three separate trainer-to-predictor update paths into a single self-contained pipeline. Deployed in production on Instagram Reels, the memory layer raises prediction coverage from 96% to 100%, improves embedding freshness from O(5 min) to O(20 s), and narrows the training-serving Normalized Entropy (NE) gap by up to 86%, yielding over 2× recall for the freshest content and a 5-6% cold start engagement lift. Because embeddings are produced during training, the system needs no separate bulk-evaluation or publish-time recomputation, cutting training-and-publish computational cost by 30% at neutral serving computational cost.