# Effective Sequence-to-Sequence Dialogue State Tracking

arXiv: https://arxiv.org/abs/2108.13990 · HTML: https://mlsanity.com/paper/2108.13990/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Efficient Dialogue State Tracking by Selectively Overwriting Memory](https://mlsanity.com/paper/1911.03906/) (2019)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://mlsanity.com/paper/1907.07421/) (2019)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [A Sequence-to-Sequence Approach to Dialogue State Tracking](https://mlsanity.com/paper/2011.09553/) (2020)
- [Pretraining Methods for Dialog Context Representation Learning](https://mlsanity.com/paper/1906.00414/) (2019)
- [Towards Universal Dialogue State Tracking](https://mlsanity.com/paper/1810.09587/) (2018)
- [A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLP](https://mlsanity.com/paper/2104.10810/) (2021)
- [Pretraining the Noisy Channel Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2103.10518/) (2021)
- [Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representation](https://mlsanity.com/paper/1806.04441/) (2018)
- [Toward Scalable Neural Dialogue State Tracking Model](https://mlsanity.com/paper/1812.00899/) (2018)

## Built on by (most cited first, 20 of 20)

- [Description-Driven Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2201.08904/) (2022)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [Speech Aware Dialog System Technology Challenge (DSTC11)](https://mlsanity.com/paper/2212.08704/) (2022)
- [Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning](https://mlsanity.com/paper/2301.10915/) (2023)
- [Beyond the Granularity: Multi-Perspective Dialogue Collaborative Selection for Dialogue State Tracking](https://mlsanity.com/paper/2205.10059/) (2022)
- [PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts](https://mlsanity.com/paper/2306.04535/) (2023)
- [On Tracking Dialogue State by Inheriting Slot Values in Mentioned Slot Pools](https://mlsanity.com/paper/2202.07156/) (2022)
- [A Multi-Task BERT Model for Schema-Guided Dialogue State Tracking](https://mlsanity.com/paper/2207.00828/) (2022)
- [Diable: Efficient Dialogue State Tracking as Operations on Tables](https://mlsanity.com/paper/2305.17020/) (2023)
- [MetaASSIST: Robust Dialogue State Tracking with Meta Learning](https://mlsanity.com/paper/2210.12397/) (2022)
- [Chain of Thought Explanation for Dialogue State Tracking](https://mlsanity.com/paper/2403.04656/) (2024)
- [What Did You Say? Task-Oriented Dialog Datasets Are Not Conversational!?](https://mlsanity.com/paper/2203.03431/) (2022)
- [Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking](https://mlsanity.com/paper/2306.09340/) (2023)
- [Agreement Tracking for Multi-Issue Negotiation Dialogues](https://mlsanity.com/paper/2307.06524/) (2023)
- [HR-Agent: A Task-Oriented Dialogue (TOD) LLM Agent Tailored for HR Applications](https://mlsanity.com/paper/2410.11239/) (2024)
- [DSTEA: Improving Dialogue State Tracking via Entity Adaptive Pre-training](https://mlsanity.com/paper/2207.03858/) (2022)
- [Understanding Emotion Valence is a Joint Deep Learning Task](https://mlsanity.com/paper/2305.17422/) (2023)
- [Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling](https://mlsanity.com/paper/2501.10316/) (2025)
- [Towards Detecting Inconsistencies in End-to-end Generated TODs](https://mlsanity.com/paper/2607.09338/) (2026)

## Abstract (arXiv)

Sequence-to-sequence models have been applied to a wide variety of NLP tasks, but how to properly use them for dialogue state tracking has not been systematically investigated. In this paper, we study this problem from the perspectives of pre-training objectives as well as the formats of context representations. We demonstrate that the choice of pre-training objective makes a significant difference to the state tracking quality. In particular, we find that masked span prediction is more effective than auto-regressive language modeling. We also explore using Pegasus, a span prediction-based pre-training objective for text summarization, for the state tracking model. We found that pre-training for the seemingly distant summarization task works surprisingly well for dialogue state tracking. In addition, we found that while recurrent state context representation works also reasonably well, the model may have a hard time recovering from earlier mistakes. We conducted experiments on the MultiWOZ 2.1-2.4, WOZ 2.0, and DSTC2 datasets with consistent observations.