mlsanityData to 7 Oct 2026

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

arXiv 2106.05555 ↗First posted 10 Jun 2021cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for benchmarking context-to-response abilities of task-oriented dialogue systems. In this work, we identify inconsistencies in data preprocessing and reporting of three corpus-based metrics used on this dataset, i.e., BLEU score and Inform & Success rates. We point out a few problems of the MultiWOZ benchmark such as unsatisfactory preprocessing, insufficient or under-specified evaluation metrics, or rigid database. We re-evaluate 7 end-to-end and 6 policy optimization models in as-fair-as-possible setups, and we show that their reported scores cannot be directly compared. To facilitate comparison of future systems, we release our stand-alone standardized evaluation scripts. We also give basic recommendations for corpus-based benchmarking in future works.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

44 of 62 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  11. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  12. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
32 more
  1. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  2. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  3. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  4. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  5. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  6. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  7. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  8. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  9. 2020BLEU might be Guilty but References are not Innocentcited by 109
  10. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  11. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  12. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  13. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  14. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  15. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  16. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  17. 2020A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learningcited by 47
  18. 2019Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attentioncited by 43
  19. 2019Structured Fusion Networks for Dialogcited by 40
  20. 2020LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimizationcited by 30
  21. 2020Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue Systemcited by 28
  22. 2020Multi-Domain Dialogue Acts and Response Co-Generationcited by 26
  23. 2019Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Modelscited by 25
  24. 2015deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targetscited by 21
  25. 2020UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialoguescited by 19
  26. 2020Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluationcited by 19
  27. 2021Domain State Tracking for a Simplified Dialogue Systemcited by 17
  28. 2019Towards Best Experiment Design for Evaluating Dialogue System Outputcited by 15
  29. 2017A Copy-Augmented Sequence-to-Sequence Architecture Gives Good Performance on Task-Oriented Dialoguecited by 14
  30. 2018Controlling Personality-Based Stylistic Variation with Neural Natural Language Generatorscited by 14
  31. 2020Conversation Learner -- A Machine Teaching Tool for Building Dialog Managers for Task-Oriented Dialog Systemscited by 9
  32. 2015Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systemscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 14; 2024: 10; 2025: 7; 2026: 1.

48 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 52
  3. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  4. 2022BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialogcited by 20
  5. 2023In-Context Learning User Simulators for Task-Oriented Dialog Systemscited by 19
  6. 2022Mars: Modeling Context & State Representations with Contrastive Learning for End-to-End Task-Oriented Dialogcited by 11
  7. 2022AnyTOD: A Programmable Task-Oriented Dialog Systemcited by 10
  8. 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisonscited by 10
  9. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  10. 2022KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learningcited by 9
  11. 2024DARD: A Multi-Agent Approach for Task-Oriented Dialog Systemscited by 9
  12. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialogcited by 8
36 more
  1. 2024LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systemscited by 7
  2. 2022A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systemscited by 6
  3. 2023DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systemscited by 6
  4. 2025The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogscited by 6
  5. 2022Enhancing Task Bot Engagement with Synthesized Open-Domain Dialogcited by 5
  6. 2024Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channelcited by 5
  7. 2022Building Markovian Generative Architectures over Pretrained LM Backbones for Efficient Task-Oriented Dialog Systemscited by 4
  8. 2023Using Textual Interface to Align External Knowledge for End-to-End Task-Oriented Dialogue Systemscited by 4
  9. 2023Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluationcited by 4
  10. 2023TOD-Flow: Modeling the Structure of Task-Oriented Dialoguescited by 4
  11. 2024Chitchat as Interference: Adding User Backstories to Task-Oriented Dialoguescited by 4
  12. 2025Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMscited by 4
  13. 2022Advancing Semi-Supervised Task Oriented Dialog Systems by JSA Learning of Discrete Latent Variable Modelscited by 3
  14. 2022Dialogue Evaluation with Offline Reinforcement Learningcited by 3
  15. 2022Learning Interpretable Latent Dialogue Actions With Less Supervisioncited by 3
  16. 2022Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?cited by 3
  17. 2023UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Promptcited by 3
  18. 2024Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialoguescited by 3
  19. 2024A Unified Approach to Emotion Detection and Task-Oriented Dialogue Modelingcited by 2
  20. 2024A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understandingcited by 2
  21. 2022UBARv2: Towards Mitigating Exposure Bias in Task-Oriented Dialogscited by 1
  22. 2022Can Current Task-oriented Dialogue Models Automate Real-world Scenarios in the Wild?cited by 1
  23. 2023A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-trainingcited by 1
  24. 2023LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systemscited by 1
  25. 2023Enhancing Task-Oriented Dialogues with Chitchat: a Comparative Study Based on Lexical Diversity and Divergencecited by 1
  26. 2024DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Modelscited by 1
  27. 2024Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systemscited by 1
  28. 2025CoDial: Interpretable Task-Oriented Dialogue Systems Through Dialogue Flow Alignmentcited by 1
  29. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1
  30. 2026MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answeringcited by 1
  31. 2022Jointly Reinforced User Simulator and Task-oriented Dialog System with Simplified Generative Architecturecited by 0
  32. 2023Curricular Transfer Learning for Sentence Encoded Taskscited by 0
  33. 2023Lexical Entrainment for Conversational Systemscited by 0
  34. 2024Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systemscited by 0
  35. 2025Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedbackcited by 0
  36. 2025Collaborative and Proactive Management of Task-Oriented Conversationscited by 0