# CollabLLM: From Passive Responders to Active Collaborators

arXiv: https://arxiv.org/abs/2502.00640 · HTML: https://mlsanity.com/paper/2502.00640/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 101 references that are in the map.
- First posted on arXiv 2 Feb 2025.

## Builds on (most cited first, 30 of 41)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023)
- [Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance](https://mlsanity.com/paper/2410.12361/) (2024)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.16745/) (2024)
- [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023)
- [Multi-turn Reinforcement Learning from Preference Human Feedback](https://mlsanity.com/paper/2405.14655/) (2024)
- [Direct Multi-Turn Preference Optimization for Language Agents](https://mlsanity.com/paper/2406.14868/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training](https://mlsanity.com/paper/2406.00222/) (2024)
- [How to Build User Simulators to Train RL-based Dialog Systems](https://mlsanity.com/paper/1909.01388/) (2019)
- [Understanding User Experience in Large Language Model Interactions](https://mlsanity.com/paper/2401.08329/) (2024)
- [Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2310.14696/) (2023)
- [Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations](https://mlsanity.com/paper/2408.15232/) (2024)
- [ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification](https://mlsanity.com/paper/2310.10996/) (2023)
- [Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations](https://mlsanity.com/paper/2311.05584/) (2023)

## Built on by (most cited first, 30 of 124)

- [VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents](https://mlsanity.com/paper/2510.16907/) (2025)
- [Training Proactive and Personalized LLM Agents](https://mlsanity.com/paper/2511.02208/) (2025)
- [Structured Uncertainty guided Clarification for LLM Agents](https://mlsanity.com/paper/2511.08798/) (2025)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [When One LLM Drools, Multi-LLM Collaboration Rules](https://mlsanity.com/paper/2502.04506/) (2025)
- [Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data](https://mlsanity.com/paper/2503.20749/) (2025)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations](https://mlsanity.com/paper/2603.01966/) (2026)
- [BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions](https://mlsanity.com/paper/2510.05318/) (2025)
- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Teaching Language Models To Gather Information Proactively](https://mlsanity.com/paper/2507.21389/) (2025)
- [When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation](https://mlsanity.com/paper/2604.00892/) (2026)
- [TOM-SWE: User Mental Modeling For Software Engineering Agents](https://mlsanity.com/paper/2510.21903/) (2025)
- [The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation](https://mlsanity.com/paper/2511.02687/) (2025)
- [ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind](https://mlsanity.com/paper/2505.22961/) (2025)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [InfoPO: Information-Driven Policy Optimization for User-Centric Agents](https://mlsanity.com/paper/2603.00656/) (2026)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Human-AI Co-design for Clinical Prediction Models](https://mlsanity.com/paper/2601.09072/) (2026)
- [Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction](https://mlsanity.com/paper/2603.04783/) (2026)
- [A Taxonomy of Prompt Defects in LLM Systems](https://mlsanity.com/paper/2509.14404/) (2025)
- [Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences](https://mlsanity.com/paper/2510.06105/) (2025)
- [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](https://mlsanity.com/paper/2601.21972/) (2026)
- [MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility](https://mlsanity.com/paper/2605.16616/) (2026)
- [MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaboration](https://mlsanity.com/paper/2601.02702/) (2026)
- [PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction](https://mlsanity.com/paper/2602.06714/) (2026)
- [Learning Next Action Predictors from Human-Computer Interaction](https://mlsanity.com/paper/2603.05923/) (2026)
- [Expert Preference-based Evaluation of Automated Related Work Generation](https://mlsanity.com/paper/2508.07955/) (2025)

## Abstract (arXiv)

Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction. As a result, they often respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inefficient conversations. To address these limitations, we introduce CollabLLM, a novel and general training framework that enhances multiturn human-LLM collaboration. Its key innovation is a collaborative simulation that estimates the long-term contribution of responses using Multiturn-aware Rewards. By reinforcement fine-tuning these rewards, CollabLLM goes beyond responding to user requests, and actively uncovers user intent and offers insightful suggestions-a key step towards more human-centered AI. We also devise a multiturn interaction benchmark with three challenging tasks such as document creation. CollabLLM significantly outperforms our baselines with averages of 18.5% higher task performance and 46.3% improved interactivity by LLM judges. Finally, we conduct a large user study with 201 judges, where CollabLLM increases user satisfaction by 17.6% and reduces user spent time by 10.4%.