mlsanityData to 14 Oct 2026

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

arXiv 2008.09335 ↗First posted 21 Aug 2020cs.CL cs.LGcited by 158

Not on the map. Why?

Abstract (arXiv)

Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.

Key facts. Cited by 158 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 25; 2022: 36; 2023: 32; 2024: 19; 2025: 23; 2026: 23.

158 papers.

Sort:

By area:

  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  2. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  3. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  4. 2022UL2: Unifying Language Learning Paradigmscited by 295
  5. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  6. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  7. 2023Compositional Exemplars for In-context Learningcited by 144
  8. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  9. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  10. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  11. 2024Making Text Embedders Few-Shot Learnerscited by 79
  12. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 70
38 more
  1. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  2. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  3. 2023Coverage-based Example Selection for In-Context Learningcited by 59
  4. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  5. 2021BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modelingcited by 56
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  7. 2021DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendationcited by 46
  8. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  9. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 39
  10. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  11. 2021Controllable Semantic Parsing via Retrieval Augmentationcited by 35
  12. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  13. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  14. 2021GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systemscited by 31
  15. 2024StructLM: Towards Building Generalist Models for Structured Knowledge Groundingcited by 29
  16. 2023Revisiting non-English Text Simplification: A Unified Multilingual Benchmarkcited by 27
  17. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
  18. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  19. 2022Cross-Lingual Dialogue Dataset Creation via Outline-Based Generationcited by 23
  20. 2022MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsingcited by 22
  21. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  22. 2022CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsingcited by 21
  23. 2023A Material Lens on Coloniality in NLPcited by 21
  24. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  25. 2021Intent Classification and Slot Filling for Privacy Policiescited by 19
  26. 2021Zero-Shot Cross-lingual Semantic Parsingcited by 19
  27. 2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19
  28. 2021Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsingcited by 18
  29. 2023CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Taskscited by 18
  30. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
  31. 2021Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Datacited by 17
  32. 2023PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogscited by 16
  33. 2024Text Clustering as Classification with LLMscited by 16
  34. 2021Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognitioncited by 15
  35. 2021RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsingcited by 15
  36. 2022Transforming Sequence Tagging Into A Seq2Seq Taskcited by 15
  37. 2022TAGPRIME: A Unified Framework for Relational Structure Extractioncited by 15
  38. 2022Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsingcited by 15

50 most cited of 158 All 158 (JSON)