# Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing

arXiv: https://arxiv.org/abs/2104.07275 · HTML: https://mlsanity.com/paper/2104.07275/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Conversational Semantic Parsing](https://mlsanity.com/paper/2009.13655/) (2020)
- [Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog](https://mlsanity.com/paper/2104.04923/) (2021)
- [Non-Autoregressive Machine Translation with Disentangled Context Transformer](https://mlsanity.com/paper/2001.05136/) (2020)
- [Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding](https://mlsanity.com/paper/2010.03714/) (2020)
- [PyText: A Seamless Path from NLP research to production](https://mlsanity.com/paper/1812.08729/) (2018)

## Built on by (most cited first, 18 of 18)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2106.11890/) (2021)
- [RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2109.10410/) (2021)
- [Chaining Simultaneous Thoughts for Numerical Reasoning](https://mlsanity.com/paper/2211.16482/) (2022)
- [Compositional Task-Oriented Parsing as Abstractive Question Answering](https://mlsanity.com/paper/2205.02068/) (2022)
- [Semantic Parsing in Task-Oriented Dialog with Recursive Insertion-based Encoder](https://mlsanity.com/paper/2109.04500/) (2021)
- [Deliberation Model for On-Device Spoken Language Understanding](https://mlsanity.com/paper/2204.01893/) (2022)
- [Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2202.00901/) (2022)
- [X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing](https://mlsanity.com/paper/2106.03777/) (2021)
- [Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning](https://mlsanity.com/paper/2204.06748/) (2022)
- [TreePiece: Faster Semantic Parsing via Tree Tokenization](https://mlsanity.com/paper/2303.17161/) (2023)
- [Towards Zero-Shot Frame Semantic Parsing with Task Agnostic Ontologies and Simple Labels](https://mlsanity.com/paper/2305.03793/) (2023)
- [Low-Resource Compositional Semantic Parsing with Concept Pretraining](https://mlsanity.com/paper/2301.09809/) (2023)
- [Assessing Data Efficiency in Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2107.04736/) (2021)
- [AutoNLU: Detecting, root-causing, and fixing NLU model errors](https://mlsanity.com/paper/2110.06384/) (2021)
- [Shift-Reduce Task-Oriented Semantic Parsing with Stack-Transformers](https://mlsanity.com/paper/2210.11984/) (2022)
- [PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2406.07823/) (2024)

## Abstract (arXiv)

An effective recipe for building seq2seq, non-autoregressive, task-oriented parsers to map utterances to semantic frames proceeds in three steps: encoding an utterance x, predicting a frame's length |y|, and decoding a |y|-sized frame with utterance and ontology tokens. Though empirically strong, these models are typically bottlenecked by length prediction, as even small inaccuracies change the syntactic and semantic characteristics of resulting frames. In our work, we propose span pointer networks, non-autoregressive parsers which shift the decoding task from text generation to span prediction; that is, when imputing utterance spans into frame slots, our model produces endpoints (e.g., [i, j]) as opposed to text (e.g., "6pm"). This natural quantization of the output space reduces the variability of gold frames, therefore improving length prediction and, ultimately, exact match. Furthermore, length prediction is now responsible for frame syntax and the decoder is responsible for frame semantics, resulting in a coarse-to-fine model. We evaluate our approach on several task-oriented semantic parsing datasets. Notably, we bridge the quality gap between non-autogressive and autoregressive parsers, achieving 87 EM on TOPv2 (Chen et al. 2020). Furthermore, due to our more consistent gold frames, we show strong improvements in model generalization in both cross-domain and cross-lingual transfer in low-resource settings. Finally, due to our diminished output vocabulary, we observe 70% reduction in latency and 83% reduction in memory at beam size 5 compared to prior non-autoregressive parsers.