# LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation

arXiv: https://arxiv.org/abs/2605.29280 · HTML: https://mlsanity.com/paper/2605.29280/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 30 of 35)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [BARS: Towards Open Benchmarking for Recommender Systems](https://mlsanity.com/paper/2205.09626/) (2022)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation](https://mlsanity.com/paper/2106.10865/) (2021)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta](https://mlsanity.com/paper/2311.09544/) (2023)
- [Privileged Features Distillation at Taobao Recommendations](https://mlsanity.com/paper/1907.05171/) (2019)
- [Toward Understanding Privileged Features Distillation in Learning-to-Rank](https://mlsanity.com/paper/2209.08754/) (2022)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model](https://mlsanity.com/paper/2405.00338/) (2024)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)

## Abstract (arXiv)

Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical RePresentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half. Through systematic experiments, LoopFM demonstrates a scaling law in sequence length, embedding dimension, and upstream FM size.