# Learning Steerable Clarification Policies with Collaborative Self-play

arXiv: https://arxiv.org/abs/2512.04068 · HTML: https://mlsanity.com/paper/2512.04068/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 30 of 31)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains](https://mlsanity.com/paper/2501.05707/) (2025)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [Selectively Answering Ambiguous Questions](https://mlsanity.com/paper/2305.14613/) (2023)
- [Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training](https://mlsanity.com/paper/2406.00222/) (2024)
- [Aligning Language Models to Explicitly Handle Ambiguity](https://mlsanity.com/paper/2404.11972/) (2024)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Training Proactive and Personalized LLM Agents](https://mlsanity.com/paper/2511.02208/) (2025)
- [PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance](https://mlsanity.com/paper/2210.08817/) (2022)
- [Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2310.14696/) (2023)
- [MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models](https://mlsanity.com/paper/2402.01620/) (2024)
- [Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning](https://mlsanity.com/paper/2407.15762/) (2024)
- [Decision-Oriented Dialogue for Human-AI Collaboration](https://mlsanity.com/paper/2305.20076/) (2023)
- [Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability](https://mlsanity.com/paper/2510.06084/) (2025)
- [Teaching Models to Balance Resisting and Accepting Persuasion](https://mlsanity.com/paper/2410.14596/) (2024)
- [Resolving Intent Ambiguities by Retrieving Discriminative Clarifying Questions](https://mlsanity.com/paper/2008.07559/) (2020)
- [Don't lie to your friends: Learning what you know from collaborative self-play](https://mlsanity.com/paper/2503.14481/) (2025)
- [Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue](https://mlsanity.com/paper/2203.07657/) (2022)
- [Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering](https://mlsanity.com/paper/2409.02361/) (2024)

## Built on by (most cited first, 5 of 5)

- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insight](https://mlsanity.com/paper/2602.15259/) (2026)
- [Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution](https://mlsanity.com/paper/2602.16154/) (2026)
- [Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generation](https://mlsanity.com/paper/2605.25831/) (2026)
- [When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain](https://mlsanity.com/paper/2609.24290/) (2026)

## Abstract (arXiv)

To handle underspecified or ambiguous queries, AI assistants need a policy for managing their uncertainty to determine (a) when to guess the user intent and answer directly, (b) when to enumerate and answer multiple possible intents, and (c) when to ask a clarifying question. However, such policies are contextually dependent on factors such as user preferences or modality. For example, enumerating multiple possible user intentions is cumbersome on small screens or in a voice setting. In this work, we propose to train steerable policies for managing this uncertainty using self-play. Given two agents, one simulating a user and the other an AI assistant, we generate conversations where the user issues a potentially ambiguous query, and the assistant needs to determine how to respond. Importantly, the model takes as input the numerical cost of each clarification question, and each generated word, and is asked to take the action that will maximize its final reward, which is the cost-penalized accuracy. We use Reinforced Self-Training (ReST) to train our model to achieve high reward and show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs, leading to higher reward and accuracy. Moreover, our procedure also generalizes to numerical cost values that were unobserved at training time.