# τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems

arXiv: https://arxiv.org/abs/2606.10156 · HTML: https://mlsanity.com/paper/2606.10156/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 21 of 21)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](https://mlsanity.com/paper/2009.14306/) (2020)
- [MACRec: a Multi-Agent Collaboration Framework for Recommendation](https://mlsanity.com/paper/2402.15235/) (2024)
- [Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2305.13112/) (2023)
- [CRSLab: An Open-Source Toolkit for Building Conversational Recommender System](https://mlsanity.com/paper/2101.00939/) (2021)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)
- [Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M](https://mlsanity.com/paper/2505.10212/) (2025)
- [RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems](https://mlsanity.com/paper/2507.22897/) (2025)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [Interactive Recommendation Agent with Active User Commands](https://mlsanity.com/paper/2509.21317/) (2025)
- [Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition](https://mlsanity.com/paper/2504.20094/) (2025)
- [Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulation](https://mlsanity.com/paper/2510.05624/) (2025)
- [SciNUP: Natural Language User Interest Profiles for Scientific Literature Recommendation](https://mlsanity.com/paper/2510.21352/) (2025)
- [Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?](https://mlsanity.com/paper/2602.13626/) (2026)
- [CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems](https://mlsanity.com/paper/2412.10514/) (2024)

## Built on by (most cited first, 3 of 3)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [MM-VeriRec: Failure-Guided Fusion for Verifiable Agentic Multimodal Recommendation](https://mlsanity.com/paper/2609.31718/) (2026)
- [Personal-Agent Mediated Recommendation with Cross-Platform User History](https://mlsanity.com/paper/2610.07588/) (2026)

## Abstract (arXiv)

As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a-judge" evaluations, which introduce subjectivity, high costs and inconsistency. We present τ-Rec, a benchmark for agentic recommender systems that replaces subjective evaluation with verifiable rewards and a reveal-tagged elicitation (RTE) mechanism that controls how task constraints surface during dialogue. By testing agents against structured catalog predicates and employing a passk reliability metric, τ-Rec provides a systematic test for consistent reasoning. Our evaluation of nine configurations across five model families -- GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B and GPT-5 mini -- reveals a steep reliability cliff, where even the best model achieves only ~57% at pass1 and ~35% at pass4, highlighting a critical gap in current conversational agent deployment. All code and data are publicly available at https://github.com/nbharaths/tau-rec.