mlsanityData to 7 Oct 2026

Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

arXiv 2302.10342 ↗First posted 20 Feb 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

When learning task-oriented dialogue (ToD) agents, reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user-specific goals. Prior works mainly focus on adopting advanced RL techniques to train the ToD agents, while the design of the reward function is not well studied. This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end (E2E) ToD agents. Specifically, we introduce two generalized objectives for reward-function learning, inspired by the classical learning-to-rank literature. Further, we utilize the learned reward function to guide the training of the E2E ToD agent. With the proposed techniques, we achieve competitive results on the E2E response-generation task on the Multiwoz 2.0 dataset. Source code and checkpoints are publicly released at https://github.com/Shentao-YANG/Fantastic_Reward_ICLR2023.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 63 of its 98 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

63 of 98 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
38 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  9. 2022Teaching language models to support answers with verified quotescited by 290
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  11. 2020Critic Regularized Regressioncited by 269
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  13. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  14. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  15. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  16. 2018Neural Approaches to Conversational AIcited by 164
  17. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  18. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  19. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  20. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  21. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  22. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  23. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  24. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  25. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  26. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  27. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  28. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  29. 2020Bayesian Attention Modulescited by 51
  30. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  31. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  32. 2019Structured Fusion Networks for Dialogcited by 40
  33. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  34. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialogcited by 37
  35. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 33
  36. 2021Annotation Inconsistency and Entity Bias in MultiWOZcited by 30
  37. 2022A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learningcited by 30
  38. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 2; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  2. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  4. 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 52
  5. 2024Planning Like Human: A Dual-process Framework for Dialogue Planningcited by 32
  6. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  7. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  8. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  9. 2024DARD: A Multi-Agent Approach for Task-Oriented Dialog Systemscited by 9
  10. 2023Enhancing Large Language Model Induced Task-Oriented Dialogue Systems Through Look-Forward Motivated Goalscited by 8
  11. 2024DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automatoncited by 5
  12. 2024Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialoguecited by 3
7 more
  1. 2023Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented Systemcited by 2
  2. 2023Exploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Trackingcited by 1
  3. 2024Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systemscited by 1
  4. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  5. 2024Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Expertscited by 0
  6. 2025Planning with Diffusion Models for Target-Oriented Dialogue Systemscited by 0
  7. 2026Categorical Internalisation of Environmental Groupoids for Generalisable POMDP Solvingcited by 0