mlsanityData to 14 Oct 2026

Tiny Neural Models for Seq2Seq

arXiv 2108.03340 ↗First posted 7 Aug 2021cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Semantic parsing models with applications in task oriented dialog systems require efficient sequence to sequence (seq2seq) architectures to be run on-device. To this end, we propose a projection based encoder-decoder model referred to as pQRNN-MAtt. Studies based on projection methods were restricted to encoder-only models, and we believe this is the first study extending it to seq2seq architectures. The resulting quantized models are less than 3.5MB in size and are well suited for on-device latency critical applications. We show that on MTOP, a challenging multilingual semantic parsing dataset, the average model performance surpasses LSTM based seq2seq model that uses pre-trained embeddings despite being 85x smaller. Furthermore, the model can be an effective student for distilling large pre-trained models such as T5/BERT.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 7 Aug 2021.

Builds on

15 of 18 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  7. 2015Pointer Networkscited by 895
  8. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  9. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  10. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  11. 2016Quasi-Recurrent Neural Networkscited by 87
  12. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
3 more
  1. 2020Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsingcited by 59
  2. 2017ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projectionscited by 11
  3. 2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing
  2. 2025TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines
  3. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inference
  4. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning
  5. 2024Evaluating Quantized Large Language Models
  6. 2023Task-agnostic Distillation of Encoder-Decoder Language Models
  7. 2025textttSEM-CTRL: Semantically Controlled Decoding
  8. 2026Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model