mlsanityData to 14 Oct 2026

KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning

arXiv 2211.16773 ↗First posted 30 Nov 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

In task-oriented dialogs (TOD), reinforcement learning (RL) algorithms train a model to directly optimize response for task-related metrics. However, RL needs to perform exploration, which can be time-consuming due to the slow auto-regressive sequence generation process. We investigate an approach to create a more efficient RL-based algorithm to improve TOD performance in an offline setting. First, we use a faster generation procedure that samples from independent next-word distributions after training the language model (LM) with supervised learning. We then introduce a fine-grained reward function to help the model focus on learning key information in a dialog, by measuring the importance and semantic closeness of each generated token. Experiments on the MultiWoZ dataset show our new training algorithm, Keywords Reinforcement Learning with Next-word Sampling (KRLS), achieves state-of-the-art performance on the end-to-end response generation task, with a 15% training time reduction compared to a standard RL algorithm using auto-regressive generation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

51 of 67 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
38 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  3. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  5. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  6. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  7. 2020A Simple Language Model for Task-Oriented Dialoguecited by 332
  8. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  10. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  12. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  13. 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 209
  14. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 204
  15. 2021Challenges in Detoxifying Language Modelscited by 169
  16. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  17. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  18. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  19. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  20. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  21. 2020Text Generation by Learning from Demonstrationscited by 64
  22. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  23. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 51
  24. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  25. 2019How to Build User Simulators to Train RL-based Dialog Systemscited by 46
  26. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  27. 2021Transferable Dialogue Systems and User Simulatorscited by 36
  28. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 34
  29. 2020LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimizationcited by 30
  30. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28
  31. 2019Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Modelscited by 25
  32. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23
  33. 2020Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decompositioncited by 23
  34. 2022BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialogcited by 20
  35. 2016Conditional Generation and Snapshot Learning in Neural Dialogue Systemscited by 17
  36. 2021Domain State Tracking for a Simplified Dialogue Systemcited by 17
  37. 2021A Student-Teacher Architecture for Dialog Domain Adaptation under the Meta-Learning Settingcited by 5
  38. 2021Causal-aware Safe Policy Improvement for Task-oriented dialoguecited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2024DARD: A Multi-Agent Approach for Task-Oriented Dialog Systemscited by 9
  2. 2023DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systemscited by 6
  3. 2024DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automatoncited by 5
  4. 2023Using Textual Interface to Align External Knowledge for End-to-End Task-Oriented Dialogue Systemscited by 4
  5. 2024Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialoguecited by 3
  6. 2025Empowering LLMs in Task-Oriented Dialogues: A Domain-Independent Multi-Agent Framework and Fine-Tuning Strategycited by 3
  7. 2025PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Personacited by 2
  8. 2026Unlocking Proactivity in Task-Oriented Dialoguecited by 1
  9. 2024Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Expertscited by 0