mlsanityData to 7 Oct 2026

Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training

arXiv 2406.00222 ↗First posted 31 May 2024cs.CL cs.AI cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

also in 1 more
Abstract (arXiv)

Large language models (LLMs), optimized through human feedback, have rapidly emerged as a leading paradigm for developing intelligent conversational assistants. However, despite their strong performance across many benchmarks, LLM-based agents might still lack conversational skills such as disambiguation -- when they are faced with ambiguity, they often overhedge or implicitly guess users' true intents rather than asking clarification questions. Under task-specific settings, high-quality conversation samples are often limited, constituting a bottleneck for LLMs' ability to learn optimal dialogue action policies. We propose Action-Based Contrastive Self-Training (ACT), a quasi-online preference optimization algorithm based on Direct Preference Optimization (DPO), that enables data-efficient dialogue policy learning in multi-turn conversation modeling. We demonstrate ACT's efficacy under in data-efficient tuning scenarios, even when there is no action label available, using multiple real-world conversational tasks: tabular-grounded question-answering, machine reading comprehension, and AmbigSQL, a novel task for disambiguating information-seeking requests for complex SQL generation towards data analysis agents. Additionally, we propose evaluating LLMs' ability to function as conversational agents by examining whether they can implicitly recognize and reason about ambiguity in conversation. ACT demonstrates substantial conversation modeling improvements over standard tuning approaches like supervised fine-tuning and DPO.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 85 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

53 of 85 references.

References · 53

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Mistral 7Bcited by 3,065
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
38 more
  1. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  11. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  13. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  14. 2024Iterative Reasoning Preference Optimizationcited by 244
  15. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  16. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  17. 2024Many-Shot In-Context Learningcited by 203
  18. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  19. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  20. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  21. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  22. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  23. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  24. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  25. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 60
  26. 2022Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planningcited by 59
  27. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  28. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  29. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  30. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  31. 2018Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systemscited by 42
  32. 2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
  33. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  34. 2019MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversationscited by 20
  35. 2016Conditional Generation and Snapshot Learning in Neural Dialogue Systemscited by 17
  36. 2023SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Datacited by 16
  37. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Modelscited by 15
  38. 2022Seamlessly Integrating Factual Information and Social Content with Persuasive Dialoguecited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 30.

53 papers.

Sort:

By area:

  1. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  2. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  3. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  4. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  5. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 34
  6. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  7. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  8. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  9. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  10. 2025Teaching Language Models To Gather Information Proactivelycited by 14
  11. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  12. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
38 more
  1. 2026Value of Information: A Framework for Human-Agent Communicationcited by 10
  2. 2025Charts Are Not Images: On the Challenges of Scientific Chart Editingcited by 9
  3. 2025DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Modelscited by 8
  4. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  5. 2025RECAP: REwriting Conversations for Intent Understanding in Agentic Planningcited by 7
  6. 2026U-Fold: Dynamic Intent-Aware Context Folding for User-Centric Agentscited by 7
  7. 2025Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Promptscited by 5
  8. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  9. 2026π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflowscited by 5
  10. 2025Zero-Shot and Efficient Clarification Need Prediction in Conversational Searchcited by 4
  11. 2025Query Understanding in LLM-based Conversational Information Seekingcited by 4
  12. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  13. 2024Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modelingcited by 3
  14. 2025Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Promptingcited by 3
  15. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  16. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 3
  17. 2026Asking What Matters: Reward-Driven Clarification for Software Engineering Taskscited by 3
  18. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  19. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  20. 2026Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insightcited by 2
  21. 2026MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homescited by 2
  22. 2026Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agentscited by 2
  23. 2025Mixed-Initiative Dialog for Human-Robot Collaborative Manipulationcited by 1
  24. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  25. 2026Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detectioncited by 1
  26. 2026IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learningcited by 1
  27. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptationcited by 1
  28. 2026Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gapcited by 1
  29. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  30. 2026Interactive Clarification for Cloud Infrastructure-as-Code Synthesiscited by 0
  31. 2026Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversationcited by 0
  32. 2026Alignment has a Fantasia Problemcited by 0
  33. 2026An Interactive Paradigm for Deep Researchcited by 0
  34. 2026Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QAcited by 0
  35. 2026TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignmentcited by 0
  36. 2026Interactive Task Alignment as a POMDPcited by 0
  37. 2026Ask or Answer: A Decision Framework for Multi-Turn Health Misinformation Interventioncited by 0
  38. 2026From Uncertainty to Clinical Risk: Severity-Aware Conformal Planning for Interactive Medical Diagnosiscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings
  2. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
  3. 2026Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
  4. 2026SOMA: Efficient Multi-turn LLM Serving via Small Language Model
  5. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
  6. 2025ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
  7. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  8. 2024Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment