# Sliding Window Training -- Utilizing Historical Recommender Systems Data for Foundation Models

arXiv: https://arxiv.org/abs/2409.14517 · HTML: https://mlsanity.com/paper/2409.14517/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 4 of 4)

- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest](https://mlsanity.com/paper/2306.00248/) (2023)

## Built on by (most cited first, 3 of 3)

- [The Value of Personalized Recommendations: Evidence from Netflix](https://mlsanity.com/paper/2511.07280/) (2025)
- [Efficient Dataset Selection for Continual Adaptation of Generative Recommenders](https://mlsanity.com/paper/2604.07739/) (2026)
- [Is Sliding Window All You Need? An Open Framework for Long-Sequence Recommendation](https://mlsanity.com/paper/2604.12372/) (2026)

## Abstract (arXiv)

Long-lived recommender systems (RecSys) often encounter lengthy user-item interaction histories that span many years. To effectively learn long term user preferences, Large RecSys foundation models (FM) need to encode this information in pretraining. Usually, this is done by either generating a long enough sequence length to take all history sequences as input at the cost of large model input dimension or by dropping some parts of the user history to accommodate model size and latency requirements on the production serving side. In this paper, we introduce a sliding window training technique to incorporate long user history sequences during training time without increasing the model input dimension. We show the quantitative & qualitative improvements this technique brings to the RecSys FM in learning user long term preferences. We additionally show that the average quality of items in the catalog learnt in pretraining also improves.