mlsanityData to 14 Oct 2026

AutoNLU: Detecting, root-causing, and fixing NLU model errors

arXiv 2110.06384 ↗First posted 12 Oct 2021cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Improving the quality of Natural Language Understanding (NLU) models, and more specifically, task-oriented semantic parsing models, in production is a cumbersome task. In this work, we present a system called AutoNLU, which we designed to scale the NLU quality improvement process. It adds automation to three key steps: detection, attribution, and correction of model errors, i.e., bugs. We detected four times more failed tasks than with random sampling, finding that even a simple active learning sampling method on an uncalibrated model is surprisingly effective for this purpose. The AutoNLU tool empowered linguists to fix ten times more semantic parsing bugs than with prior manual processes, auto-correcting 65% of all identified bugs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

16 of 27 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  4. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  5. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  7. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  8. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  9. 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
  10. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  11. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
  12. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
4 more
  1. 2021Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsingcited by 18
  2. 2017SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctnesscited by 5
  3. 2021Diagnosing Transformers in Task-Oriented Semantic Parsingcited by 5
  4. 2019Active Learning for Domain Classification in a Commercial Spoken Personal Assistantcited by 3

Built on by

Not cited yet.