mlsanityData to 14 Oct 2026

On the Effectiveness of Offline RL for Dialogue Response Generation

arXiv 2307.12425 ↗First posted 23 Jul 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

A common training technique for language models is teacher forcing (TF). TF attempts to match human language exactly, even though identical meanings can be expressed in different ways. This motivates use of sequence-level objectives for dialogue response generation. In this paper, we study the efficacy of various offline reinforcement learning (RL) methods to maximize such objectives. We present a comprehensive evaluation across multiple datasets, models, and metrics. Offline RL shows a clear performance improvement over teacher forcing while not inducing training instability or sacrificing practical training budgets.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 109 references that are in the map. First posted on arXiv 23 Jul 2023.

Builds on

36 of 109 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
24 more
  1. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  2. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  5. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  6. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 204
  7. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 140
  8. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  9. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  10. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  11. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  12. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 75
  13. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  14. 2020Text Generation by Learning from Demonstrationscited by 64
  15. 2020Improving Dialog Systems for Negotiation with Personality Modelingcited by 38
  16. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 34
  17. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 33
  18. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  19. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23
  20. 2017Adversarial Neural Machine Translationcited by 21
  21. 2021Amendable Generation for Dialogue State Trackingcited by 21
  22. 2022Long-term Control for Dialogue Generation: Methods and Evaluationcited by 5
  23. 2019Building a Production Model for Retrieval-Based Chatbotscited by 4
  24. 2022lilGym: Natural Language Visual Reasoning with Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 1; 2026: 1.

6 papers.

  1. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10
  2. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Spacecited by 8
  3. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  4. 2026ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agentscited by 3
  5. 2023Workflow-Guided Response Generation for Task-Oriented Dialoguecited by 1
  6. 2024Provably Adaptive Average Reward Reinforcement Learning for Metric Spacescited by 0