mlsanityData to 7 Oct 2026

Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner

arXiv 2406.11978 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 12

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We present an approach called Dialogue Action Tokens (DAT) that adapts language model agents to plan goal-directed dialogues. The core idea is to treat each utterance as an action, thereby converting dialogues into games where existing approaches such as reinforcement learning can be applied. Specifically, we freeze a pretrained language model and train a small planner model that predicts a continuous action vector, used for controlled generation in each round. This design avoids the problem of language degradation under reward optimization. When evaluated on the Sotopia platform for social simulations, the DAT-steered LLaMA model surpasses GPT-4's performance. We also apply DAT to steer an attacker language model in a novel multi-turn red-teaming setting, revealing a potential new attack surface.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 51 of its 63 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

51 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct

Other references · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
38 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  7. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2023Reward Design with Language Modelscited by 257
  13. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  14. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  15. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  16. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  17. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  18. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  19. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  20. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  21. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  22. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  23. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  24. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  25. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 93
  26. 2022Evaluating Human-Language Model Interactioncited by 90
  27. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  28. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 73
  29. 2019Target-Guided Open-Domain Conversationcited by 66
  30. 2019Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialoguecited by 44
  31. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  32. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  33. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 33
  34. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  35. 2022Task Ambiguity in Humans and Language Modelscited by 27
  36. 2021Control Prefixes for Parameter-Efficient Text Generationcited by 23
  37. 2024Social Skill Training with Large Language Modelscited by 13
  38. 2024Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Futurecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 6.

12 papers.

  1. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  2. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  3. 2025Adaptive Social Learning via Mode Policy Optimization for Language Agentscited by 15
  4. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  5. 2025EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learningcited by 8
  6. 2025Convert Language Model into a Value-based Strategic Plannercited by 4
  7. 2026SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attributioncited by 3
  8. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  9. 2026Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewardingcited by 0
  10. 2026ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agentscited by 0
  11. 2026Proactive Dialogue Policy Optimization via Cognitive-State Transitioncited by 0
  12. 2026RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  2. 2026LinguaGame: A Linguistically Grounded Game-Theoretic Paradigm for Multi-Agent Dialogue Generation
  3. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
  4. 2026One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent
  5. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits
  6. 2026First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents
  7. 2025MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Models
  8. 2025PRINCIPLES: Synthetic Strategy Memory for Proactive Dialogue Agents