# RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents

arXiv: https://arxiv.org/abs/2605.18805 · HTML: https://mlsanity.com/paper/2605.18805/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 30 of 31)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Explainable Recommendation: A Survey and New Perspectives](https://mlsanity.com/paper/1804.11192/) (2018)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Inferring Networks of Substitutable and Complementary Products](https://mlsanity.com/paper/1506.08839/) (2015)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learning](https://mlsanity.com/paper/2405.15114/) (2024)
- [A Survey on LLM-powered Agents for Recommender Systems](https://mlsanity.com/paper/2502.10050/) (2025)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [KuaiSim: A Comprehensive Simulator for Recommender Systems](https://mlsanity.com/paper/2309.12645/) (2023)
- [ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents](https://mlsanity.com/paper/2508.04266/) (2025)
- [Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2503.16734/) (2025)
- [Bundle Recommendation with Graph Convolutional Networks](https://mlsanity.com/paper/2005.03475/) (2020)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models](https://mlsanity.com/paper/2410.20745/) (2024)
- [WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents](https://mlsanity.com/paper/2508.13024/) (2025)
- [Generative Slate Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2301.08632/) (2023)
- [ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?](https://mlsanity.com/paper/2511.22978/) (2025)

## Abstract (arXiv)

LLM recommendation agents increasingly produce structured recommendation reports: sets of items accompanied by natural-language justifications. Yet existing evaluations often reduce this setting to reranking small shortlisted candidate sets or judge reports mainly by semantic plausibility. We introduce Recommendation Atlas (Agentic Tool-Level Assessment for Shopping), or RecoAtlas, a benchmark and toolkit for evaluating shopping agents with behavior-grounded metrics. RecoAtlas complements held-out interaction metrics with learned utility proxies for relevance, complementarity, and diversity derived from interaction data, while separately measuring semantic coherence and explanation quality. Its controlled tool environment exposes agents to either semantic, behavior-aligned, or faulty tools, enabling diagnosis of whether performance gains arise from stronger reasoning, better signals, or more effective tool-use policies. Across controlled experiments, we show that RecoAtlas exhibits key properties of a meaningful benchmark for agentic systems: performance scales with model capacity and test-time compute, improves with stronger and better-aligned tools, degrades under noisy or misaligned signals, and reveals that semantic plausibility does not necessarily capture behavior-grounded utility. RecoAtlas provides a foundation for developing and evaluating shopping assistants that optimize not only for plausible recommendations, but also for coherent, behaviorally grounded recommendation sets.