mlsanityData to 14 Oct 2026

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

arXiv 2110.06800 ↗First posted 13 Oct 2021cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 34 of its 39 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

34 of 39 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  10. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  12. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
22 more
  1. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  2. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 143
  3. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 108
  4. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  5. 2018Sequence-to-Sequence Data Augmentation for Dialogue Language Understandingcited by 68
  6. 2020Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Trackingcited by 65
  7. 2021Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Trackingcited by 61
  8. 2020STAR: A Schema-Guided Dialog Dataset for Transfer Learningcited by 44
  9. 2017Towards Zero-Shot Frame Semantic Parsing for Domain Scalingcited by 40
  10. 2019Robust Zero-Shot Cross-Domain Slot Filling with Example Valuescited by 38
  11. 2021Zero-shot Generalization in Dialog State Tracking through Generative Question Answeringcited by 33
  12. 2020Schema-Guided Dialogue State Tracking Task at DSTC8cited by 32
  13. 2020Robustness Testing of Language Understanding in Task-Oriented Dialogcited by 32
  14. 2020MA-DST: Multi-Attention Based Scalable Dialog State Trackingcited by 24
  15. 2018Data Augmentation for Spoken Language Understanding via Joint Variational Generationcited by 21
  16. 2019An End-to-End Dialogue State Tracking System with Machine Reading Comprehension and Wide & Deep Classificationcited by 20
  17. 2019Improving Robustness of Task Oriented Dialog Systemscited by 15
  18. 2021Schema-Guided Paradigm for Zero-Shot Dialogcited by 14
  19. 2020Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Trackingcited by 11
  20. 2020A Fast and Robust BERT-based Dialogue State Tracker for Schema-Guided Dialogue Datasetcited by 8
  21. 2020The SPPD System for Schema Guided Dialogue State Tracking Challengecited by 5
  22. 2021SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Servicescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 11; 2024: 9; 2025: 3; 2026: 5.

34 papers.

Sort:

By area:

  1. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  2. 2022GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformerscited by 23
  3. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  4. 2024LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational Historycited by 19
  5. 2023Towards Better Evaluation of Instruction-Following: A Case-Study in Summarizationcited by 18
  6. 2024MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Modelscited by 17
  7. 2022Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialoguecited by 16
  8. 2024HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agentcited by 12
  9. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 9
  10. 2023Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Functioncited by 7
  11. 2024Are LLMs Robust for Spoken Dialogues?cited by 7
  12. 2023The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intentscited by 6
22 more
  1. 2022Information Extraction and Human-Robot Dialogue towards Real-life Tasks: A Baseline Study with the MobileCS Datasetcited by 4
  2. 2024Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varietiescited by 4
  3. 2024Transforming Slot Schema Induction with Generative Dialogue State Inferencecited by 3
  4. 2023More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Rankingcited by 2
  5. 2023Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Trackingcited by 2
  6. 2023Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented Systemcited by 2
  7. 2025Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactionscited by 2
  8. 2023Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue Response Generationcited by 1
  9. 2023Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Inductioncited by 1
  10. 2023Grounding Description-Driven Dialogue State Trackers with Knowledge-Seeking Turnscited by 1
  11. 2024Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agentscited by 1
  12. 2024An Approach to Build Zero-Shot Slot-Filling System for Industry-Grade Conversational Assistantscited by 1
  13. 2024HR-Agent: A Task-Oriented Dialogue (TOD) LLM Agent Tailored for HR Applicationscited by 1
  14. 2025PyTOD: Programmable Task-Oriented Dialogue with Execution Feedbackcited by 1
  15. 2026The Convergence of Schema-Guided Dialogue Systems and the Model Context Protocolcited by 1
  16. 2023Hexa: Self-Improving for Knowledge-Grounded Dialogue Systemcited by 0
  17. 2023Step by Step to Fairness: Attributing Societal Bias in Task-oriented Dialogue Systemscited by 0
  18. 2025Spec-TOD: A Specialized Instruction-Tuned LLM Framework for Efficient Task-Oriented Dialogue Systemscited by 0
  19. 2026ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applicationscited by 0
  20. 2026When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systemscited by 0
  21. 2026SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agentscited by 0
  22. 2026Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backendscited by 0