# Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M

arXiv: https://arxiv.org/abs/2505.10212 · HTML: https://mlsanity.com/paper/2505.10212/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [Embarrassingly Shallow Autoencoders for Sparse Data](https://mlsanity.com/paper/1905.03375/) (2019)
- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)
- [Collaborative Large Language Model for Recommender Systems](https://mlsanity.com/paper/2311.01343/) (2023)
- [User-centered Evaluation of Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2103.06364/) (2021)
- [Large Language Models for Intent-Driven Session Recommendations](https://mlsanity.com/paper/2312.07552/) (2023)
- [Evaluating ChatGPT as a Recommender System: A Rigorous Approach](https://mlsanity.com/paper/2309.03613/) (2023)
- [Large Language Models as Data Augmenters for Cold-Start Item Recommendation](https://mlsanity.com/paper/2402.11724/) (2024)
- [FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction](https://mlsanity.com/paper/2310.19453/) (2023)
- [From Clicks to Carbon: The Environmental Toll of Recommender Systems](https://mlsanity.com/paper/2408.08203/) (2024)
- [Hyperband-based Bayesian Optimization for Black-box Prompt Selection](https://mlsanity.com/paper/2412.07820/) (2024)

## Built on by (most cited first, 12 of 12)

- [LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing](https://mlsanity.com/paper/2505.16491/) (2025)
- [When Large Language Models Know the Table: A Framework for Assessing Data Contamination in Tabular Datasets](https://mlsanity.com/paper/2510.20351/) (2025)
- [τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems](https://mlsanity.com/paper/2606.10156/) (2026)
- [Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Search](https://mlsanity.com/paper/2507.05006/) (2025)
- [Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?](https://mlsanity.com/paper/2602.13626/) (2026)
- [We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Later](https://mlsanity.com/paper/2509.09414/) (2025)
- [Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs](https://mlsanity.com/paper/2601.10496/) (2026)
- [Membership Inference Attacks on In-Context Examples in LLM-based Recommender Systems](https://mlsanity.com/paper/2508.18665/) (2025)
- [Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations](https://mlsanity.com/paper/2601.01997/) (2026)
- [Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models](https://mlsanity.com/paper/2601.02002/) (2026)
- [Fair Agents: Balancing Multistakeholder Alignment in Multi-Agent Personalization Systems](https://mlsanity.com/paper/2605.02379/) (2026)
- [Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation](https://mlsanity.com/paper/2609.08943/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become increasingly central to recommendation scenarios due to their remarkable natural language understanding and generation capabilities. Although significant research has explored the use of LLMs for various recommendation tasks, little effort has been dedicated to verifying whether they have memorized public recommendation dataset as part of their training data. This is undesirable because memorization reduces the generalizability of research findings, as benchmarking on memorized datasets does not guarantee generalization to unseen datasets. Furthermore, memorization can amplify biases, for example, some popular items may be recommended more frequently than others. In this work, we investigate whether LLMs have memorized public recommendation datasets. Specifically, we examine two model families (GPT and Llama) across multiple sizes, focusing on one of the most widely used dataset in recommender systems: MovieLens-1M. First, we define dataset memorization as the extent to which item attributes, user profiles, and user-item interactions can be retrieved by prompting the LLMs. Second, we analyze the impact of memorization on recommendation performance. Lastly, we examine whether memorization varies across model families and model sizes. Our results reveal that all models exhibit some degree of memorization of MovieLens-1M, and that recommendation performance is related to the extent of memorization. We have made all the code publicly available at: https://github.com/sisinflab/LLM-MemoryInspector