# Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agents

arXiv: https://arxiv.org/abs/2607.13418 · HTML: https://mlsanity.com/paper/2607.13418/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 26 of 26)

- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms](https://mlsanity.com/paper/2011.01731/) (2020)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems](https://mlsanity.com/paper/2002.09102/) (2020)
- [Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System](https://mlsanity.com/paper/2010.15363/) (2020)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022)
- [Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity](https://mlsanity.com/paper/2602.03794/) (2026)
- [Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs](https://mlsanity.com/paper/2510.11062/) (2025)
- [User-controllable Recommendation Against Filter Bubbles](https://mlsanity.com/paper/2204.13844/) (2022)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [When Search Meets Recommendation: Learning Disentangled Search Representation for Recommendation](https://mlsanity.com/paper/2305.10822/) (2023)
- [How the Design of YouTube Influences User Sense of Agency](https://mlsanity.com/paper/2101.11778/) (2021)
- [A Comprehensive Survey of Evaluation Techniques for Recommendation Systems](https://mlsanity.com/paper/2312.16015/) (2023)
- [A Survey of Controllable Learning: Methods and Applications in Information Retrieval](https://mlsanity.com/paper/2407.06083/) (2024)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [Multi-Agent Collaborative Filtering: Orchestrating Users and Items for Agentic Recommendations](https://mlsanity.com/paper/2511.18413/) (2025)
- [Multi-agent Attacks for Black-box Social Recommendations](https://mlsanity.com/paper/2311.07127/) (2023)
- [Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition](https://mlsanity.com/paper/2504.20094/) (2025)
- [User-Controllable Recommendation via Counterfactual Retrospective and Prospective Explanations](https://mlsanity.com/paper/2308.00894/) (2023)

## Abstract (arXiv)

Recommender systems operate as Black-Boxes, leaving users and regulators unable to steer their outputs toward specific intentions or audit their behavior. This lack of controllability, defined as the system's ability to respond to explicit guidance, remains an unaddressed dimension in existing evaluation paradigms. To fill this gap, we propose CtrlBench-Rec, a collaborative multi-agent framework for systematic assessment of controllability. We formalize three fundamental tasks: target content discovery, interest profile shaping, and popularity bias mitigation, which together measure steerability from explicit commands to implicit representation steering and finally to overcoming algorithmic biases.Extensive experiments on real-world datasets and multiple recommendation models demonstrate that our framework effectively quantifies controllability and exposes critical system bottlenecks, most notably persistent resistance to guiding long tail content. CtrlBench-Rec provides the first standardized toolkit for controllable recommendation research, algorithmic auditing, and user empowerment. Our code is released on https://github.com/caskcsg/CtrlBenchRec.