User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs
Not on the map. Why?
Abstract (arXiv)
Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks. Instead, we propose user behavior prediction, also a key aspect of personalization, as a theoretically sound, scalable, and robust alternative. We introduce a novel framework for this approach and test it on movie and music recommendation datasets for GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. Results align with our framework's predictions, showing GPT-4o outperforms GPT-4o-mini and Llama, though all models have much room for improvement, especially Llama.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 62 references that are in the map. First posted on arXiv 30 Jun 2025.
Builds on
29 of 62 references.
References · 29
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
- 2023A Survey on Large Language Models for Recommendationcited by 379
- 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
- 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
17 more
- 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
- 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 193
- 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
- 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 161
- 2024Investigating Cultural Alignment of Large Language Modelscited by 143
- 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
- 2024Personalization of Large Language Models: A Surveycited by 104
- 2023Aligning Language Models to User Opinionscited by 99
- 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
- 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
- 2023Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMscited by 64
- 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
- 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
- 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
- 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 25
- 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
- 2023Measuring the Effect of Influential Messages on Varying Personascited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.