mlsanityData to 7 Oct 2026

NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation

arXiv 2111.05196 ↗First posted 9 Nov 2021cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

35 of 65 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  5. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  6. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  7. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  11. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
23 more
  1. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  2. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  3. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  5. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  6. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  7. 2019BERT for Joint Intent Classification and Slot Fillingcited by 195
  8. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  9. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  10. 2019A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understandingcited by 106
  11. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 85
  12. 2020An Analysis of Simple Data Augmentation for Named Entity Recognitioncited by 75
  13. 2018A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Fillingcited by 53
  14. 2020CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackerscited by 43
  15. 2021Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Fillingcited by 25
  16. 2020Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Studycited by 19
  17. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  18. 2020On the Robustness of Language Encoders against Grammatical Errorscited by 18
  19. 2020What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?cited by 17
  20. 2020Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classificationcited by 10
  21. 2020Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structurescited by 10
  22. 2018Improving the Robustness of Speech Translationcited by 7
  23. 2020A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?cited by 6

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  2. 2024CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systemscited by 0