mlsanityData to 14 Oct 2026

DS-TOD: Efficient Domain Specialization for Task Oriented Dialog

arXiv 2110.08395 ↗First posted 15 Oct 2021cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD). These approaches, however, exploit general dialogic corpora (e.g., Reddit) and thus presumably fail to reliably embed domain-specific knowledge useful for concrete downstream TOD domains. In this work, we investigate the effects of domain specialization of pretrained language models (PLMs) for TOD. Within our DS-TOD framework, we first automatically extract salient domain-specific terms, and then use them to construct DomainCC and DomainReddit -- resources that we leverage for domain-specific pretraining, based on (i) masked language modeling (MLM) and (ii) response selection (RS) objectives, respectively. We further propose a resource-efficient and modular domain specialization by means of domain adapters -- additional parameter-light layers in which we encode the domain knowledge. Our experiments with prominent TOD tasks -- dialog state tracking (DST) and response retrieval (RR) -- encompassing five domains from the MultiWOZ benchmark demonstrate the effectiveness of DS-TOD. Moreover, we show that the light-weight adapter-based specialization (1) performs comparably to full fine-tuning in single domain setups and (2) is particularly suitable for multi-domain specialization, where besides advantageous computational footprint, it can offer better TOD performance.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

42 of 59 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
30 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  2. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  4. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  5. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  6. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  7. 2020A Simple Language Model for Task-Oriented Dialoguecited by 332
  8. 2020The Pushshift Reddit Datasetcited by 292
  9. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  10. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  11. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  12. 2018Neural Approaches to Conversational AIcited by 165
  13. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  14. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  15. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  16. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 143
  17. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  18. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  19. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  20. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  21. 2021Sustainable Modular Debiasing of Language Modelscited by 100
  22. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  23. 2019Training Neural Response Selection for Task-Oriented Dialogue Systemscited by 49
  24. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  25. 2019A Repository of Conversational Datasetscited by 45
  26. 2020Robustness Testing of Language Understanding in Task-Oriented Dialogcited by 32
  27. 2018Large-Scale Multi-Domain Belief Tracking with Knowledge Sharingcited by 29
  28. 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
  29. 2019PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Bookingcited by 7
  30. 2021Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Taskscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 10; 2023: 5; 2024: 6.

21 papers.

Sort:

  1. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  2. 2022Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialogcited by 30
  3. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  4. 2023Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttalscited by 18
  5. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  6. 2022Parameter-Efficient Abstractive Question Answering over Tables or Textcited by 11
  7. 2022Improved and Efficient Conversational Slot Labeling through Question Answeringcited by 10
  8. 2022Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformerscited by 10
  9. 2023TADA: Efficient Task-Agnostic Domain Adaptation for Transformerscited by 6
  10. 2024ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Fillercited by 5
  11. 2023Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detectioncited by 4
  12. 2024What the Weight?! A Unified Framework for Zero-Shot Knowledge Compositioncited by 4
9 more
  1. 2024News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendationcited by 4
  2. 2022Back to the Future: On Potential Histories in NLPcited by 3
  3. 2023SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLUcited by 3
  4. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  5. 2023Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extractioncited by 1
  6. 2022Massively Multilingual Lexical Specialization of Multilingual Transformerscited by 0
  7. 2022On the Limitations of Sociodemographic Adaptation with Transformerscited by 0
  8. 2024Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Trackingcited by 0
  9. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0