# Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale

arXiv: https://arxiv.org/abs/2604.24806 · HTML: https://mlsanity.com/paper/2604.24806/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders](https://mlsanity.com/paper/2505.04421/) (2025)
- [End-to-End User Behavior Retrieval in Click-Through RatePrediction Model](https://mlsanity.com/paper/2108.04468/) (2021)
- [tf.data: A Machine Learning Data Processing Framework](https://mlsanity.com/paper/2101.12127/) (2021)
- [TWIN V2: Scaling Ultra-Long User Behavior Sequence Modeling for Enhanced CTR Prediction at Kuaishou](https://mlsanity.com/paper/2407.16357/) (2024)
- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders](https://mlsanity.com/paper/2510.22049/) (2025)
- [DV365: Extremely Long User History Modeling at Instagram](https://mlsanity.com/paper/2506.00450/) (2025)
- [Request-Only Optimization for Recommendation Systems](https://mlsanity.com/paper/2508.05640/) (2025)
- [Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment](https://mlsanity.com/paper/2508.02929/) (2025)
- [Bending the Scaling Law Curve in Large-Scale Recommendation Systems](https://mlsanity.com/paper/2602.16986/) (2026)
- [User Long-Term Multi-Interest Retrieval Model for Recommendation](https://mlsanity.com/paper/2507.10097/) (2025)
- [LongRetriever: Towards Ultra-Long Sequence based Candidate Retrieval for Recommendation](https://mlsanity.com/paper/2508.15486/) (2025)
- [Bullion: A Column Store for Machine Learning](https://mlsanity.com/paper/2404.08901/) (2024)

## Abstract (arXiv)

Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws with sequence length, driving the frontier toward ultra-long User Interaction History (UIH). However, the industry-standard "Fat Row" paradigm, which pre-materializes these sequences into every training example, creates a storage and I/O wall where data infrastructure usage exceeds GPU training capacity due to data redundancy that is amplified in multi-tenant environments where models with vastly different sequence length requirements share a union dataset. We present a versioned late materialization paradigm that eliminates this redundancy by storing UIH once in a normalized, immutable tier and reconstructing sequences just-in-time during training via lightweight versioned pointers. The system ensures Online-to-Offline (O2O) consistency through a bifurcated protocol that prevents future leakage across both streaming and batch training, while a read-optimized immutable storage layer provides multi-dimensional projection pushdown for heterogeneous model tenants. Disaggregated data preprocessing with pipelined I/O prefetching and data-affinity optimizations masks the latency of training-time sequence reconstruction, keeping training throughput compute-bound by GPUs. Deployed on production DLRMs, the system reduces training data infrastructure resource usage while enabling aggressive sequence length scaling that delivers significant model quality gains, serving as the foundational data infrastructure for modern recommendation model architectures, including HSTU and ULTRA-HSTU.