mlsanityData to 14 Oct 2026

τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems

arXiv 2606.10156 ↗First posted 8 Jun 2026cs.IR cs.AI cs.CLcited by 3

On the map: LLM agents and tool use → Agentic recommender systems

Abstract (arXiv)

As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a-judge" evaluations, which introduce subjectivity, high costs and inconsistency. We present τ-Rec, a benchmark for agentic recommender systems that replaces subjective evaluation with verifiable rewards and a reveal-tagged elicitation (RTE) mechanism that controls how task constraints surface during dialogue. By testing agents against structured catalog predicates and employing a passk reliability metric, τ-Rec provides a systematic test for consistent reasoning. Our evaluation of nine configurations across five model families -- GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B and GPT-5 mini -- reveals a steep reliability cliff, where even the best model achieves only ~57% at pass1 and ~35% at pass4, highlighting a critical gap in current conversational agent deployment. All code and data are publicly available at https://github.com/nbharaths/tau-rec.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 21 of its 26 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

21 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems26 May
  2. 2025Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition26 Apr
  3. 2024MACRec: a Multi-Agent Collaboration Framework for Recommendation23 Feb
  4. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations31 Aug
  5. 2023RecMind: Large Language Model Powered Agent For Recommendation28 Aug

Other references · 16

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  5. 2020Towards Conversational Recommendation over Multi-Type Dialogscited by 107
  6. 2020INSPIRED: Toward Sociable Recommendation Dialog Systemscited by 93
  7. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  8. 2021CRSLab: An Open-Source Toolkit for Building Conversational Recommender Systemcited by 34
  9. 2025Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1Mcited by 12
  10. 2025RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systemscited by 12
  11. 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 12
  12. 2025Interactive Recommendation Agent with Active User Commandscited by 10
4 more
  1. 2025Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulationcited by 2
  2. 2025SciNUP: Natural Language User Interest Profiles for Scientific Literature Recommendationcited by 2
  3. 2026Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?cited by 2
  4. 2024CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  2. 2026MM-VeriRec: Failure-Guided Fusion for Verifiable Agentic Multimodal Recommendationcited by 0
  3. 2026Personal-Agent Mediated Recommendation with Cross-Platform User Historycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems
  2. 2026RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
  3. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
  4. 2026HARPO: Hierarchical Agentic Reasoning for User-Aligned Conversational Recommendation
  5. 2026Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems
  6. 2025Evaluating Conversational Recommender Systems via Large Language Models: A User-Centric Framework
  7. 2026RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation
  8. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systems