# Less is More: Benchmarking LLM Based Recommendation Agents

arXiv: https://arxiv.org/abs/2601.20316 · HTML: https://mlsanity.com/paper/2601.20316/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 10 of 10)

- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [Retrieval meets Long Context Large Language Models](https://mlsanity.com/paper/2310.03025/) (2023)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives](https://mlsanity.com/paper/2503.11924/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed for personalized product recommendations, with practitioners commonly assuming that longer user purchase histories lead to better predictions. We challenge this assumption through a systematic benchmark of four state of the art LLMs GPT-4o-mini, DeepSeek-V3, Qwen2.5-72B, and Gemini 2.5 Flash across context lengths ranging from 5 to 50 items using the REGEN dataset. Surprisingly, our experiments with 50 users in a within subject design reveal no significant quality improvement with increased context length. Quality scores remain flat across all conditions (0.17--0.23). Our findings have significant practical implications: practitioners can reduce inference costs by approximately 88% by using context (5--10 items) instead of longer histories (50 items), without sacrificing recommendation quality. We also analyze latency patterns across providers and find model specific behaviors that inform deployment decisions. This work challenges the existing ``more context is better'' paradigm and provides actionable guidelines for cost effective LLM based recommendation systems.