mlsanityData to 7 Oct 2026

Pretraining the Noisy Channel Model for Task-Oriented Dialogue

arXiv 2103.10518 ↗First posted 18 Mar 2021cs.CL cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Direct decoding for task-oriented dialogue is known to suffer from the explaining-away effect, manifested in models that prefer short and generic responses. Here we argue for the use of Bayes' theorem to factorize the dialogue task into two models, the distribution of the context given the response, and the prior for the response itself. This approach, an instantiation of the noisy channel model, both mitigates the explaining-away effect and allows the principled incorporation of large pretrained models for the response prior. We present extensive experiments showing that a noisy channel model decodes better responses compared to direct decoding and that a two stage pretraining strategy, employing both open-domain and task-oriented dialogue data, improves over randomly initialized models.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 50 of its 73 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

50 of 73 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2018Hierarchical Neural Story Generationcited by 1,182
  11. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
38 more
  1. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  3. 2020Recipes for building an open-domain chatbotcited by 593
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  5. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  6. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  7. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  8. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  9. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  10. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  11. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  12. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  13. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  14. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  15. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  16. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  17. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  18. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  19. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  20. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  21. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  22. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  23. 2020A Controllable Model of Grounded Response Generationcited by 49
  24. 2019Training Neural Response Selection for Task-Oriented Dialogue Systemscited by 48
  25. 2019Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attentioncited by 43
  26. 2019Structured Fusion Networks for Dialogcited by 40
  27. 2019Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answeringcited by 39
  28. 2019Simple and Effective Noisy Channel Modeling for Neural Machine Translationcited by 37
  29. 2019Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Modelscited by 25
  30. 2017Scalable Multi-Domain Dialogue State Trackingcited by 22
  31. 2016The Neural Noisy Channelcited by 19
  32. 2017Latent Intention Dialogue Modelscited by 15
  33. 2018Toward Scalable Neural Dialogue State Tracking Modelcited by 11
  34. 2020A Generative Model for Joint Natural Language Understanding and Generationcited by 9
  35. 2019Teacher-Student Framework Enhanced Multi-domain Dialogue Generationcited by 8
  36. 2019Neural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge Reasoningcited by 5
  37. 2019A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Expertscited by 4
  38. 2020Diversifying Task-oriented Dialogue Response Generation with Prototype Guided Paraphrasingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 6; 2024: 3; 2025: 0; 2026: 1.

23 papers.

Sort:

By area:

  1. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  2. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  3. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  4. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systemscited by 38
  5. 2022A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systemscited by 32
  6. 2022Relational Memory Augmented Language Modelscited by 24
  7. 2021DS-TOD: Efficient Domain Specialization for Task Oriented Dialogcited by 21
  8. 2021Effective Sequence-to-Sequence Dialogue State Trackingcited by 20
  9. 2023Dialogue Planning via Brownian Bridge Stochastic Process for Goal-directed Proactive Dialoguecited by 14
  10. 2022Q-TOD: A Query-driven Task-oriented Dialogue Systemcited by 10
  11. 2024Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialoguecited by 6
  12. 2021Adapting Document-Grounded Dialog Systems to Spoken Conversations using Data Augmentation and a Noisy Channel Modelcited by 5
11 more
  1. 2024Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channelcited by 5
  2. 2022OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialoguecited by 4
  3. 2023Task-oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10cited by 4
  4. 2023OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue Systemcited by 4
  5. 2022DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learningcited by 3
  6. 2022Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Modelcited by 3
  7. 2023Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systemscited by 3
  8. 2023SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLUcited by 3
  9. 2023Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented Systemcited by 2
  10. 2022Enabling arbitrary translation objectives with Adaptive Tree Searchcited by 0
  11. 2026ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modelingcited by 0