mlsanityData to 14 Oct 2026

Benchmarking Natural Language Understanding Services for building Conversational Agents

arXiv 1903.05566 ↗First posted 13 Mar 2019cs.CL cs.LGcited by 166

Not on the map. Why?

Abstract (arXiv)

We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 41; 2023: 37; 2024: 30; 2025: 16; 2026: 19.

166 papers.

Sort:

By area:

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  2. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  3. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  4. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  5. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  7. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  8. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  9. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  10. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  11. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  12. 2022Parallel Context Windows for Large Language Modelscited by 81
38 more
  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  2. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  3. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  4. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  5. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  6. 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
  7. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  8. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  9. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  10. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  11. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  12. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  13. 2023In-Context Learning for Text Classification with Many Labelscited by 35
  14. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  15. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  16. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  17. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  18. 2021An Efficient DP-SGD Mechanism for Large Scale NLP Modelscited by 27
  19. 2021Effectiveness of Pre-training for Few-shot Intent Classificationcited by 27
  20. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 26
  21. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  22. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  23. 2023Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Informationcited by 24
  24. 2022Cross-Lingual Dialogue Dataset Creation via Outline-Based Generationcited by 23
  25. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 22
  26. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  27. 2022Canary Extraction in Natural Language Understanding Modelscited by 21
  28. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  29. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  30. 2021ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learningcited by 18
  31. 2022LAD: Language Models as Data for Zero-Shot Dialogcited by 18
  32. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
  33. 2022Learning Dialogue Representations from Consecutive Utterancescited by 17
  34. 2023ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classificationcited by 17
  35. 2022Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue Systemcited by 16
  36. 2022A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialogcited by 14
  37. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  38. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 14

50 most cited of 166 All 166 (JSON)