Who Needs Decoders? Efficient Estimation of Sequence-level Attributes
Not on the map. Why?
Abstract (arXiv)
State-of-the-art sequence-to-sequence models often require autoregressive decoding, which can be highly expensive. However, for some downstream tasks such as out-of-distribution (OOD) detection and resource allocation, the actual decoding output is not needed just a scalar attribute of this sequence. In these scenarios, where for example knowing the quality of a system's output to predict poor performance prevails over knowing the output itself, is it possible to bypass the autoregressive decoding? We propose Non-Autoregressive Proxy (NAP) models that can efficiently predict general scalar-valued sequence-level attributes. Importantly, NAPs predict these metrics directly from the encodings, avoiding the expensive autoregressive decoding stage. We consider two sequence-to-sequence task: Machine Translation (MT); and Automatic Speech Recognition (ASR). In OOD for MT, NAPs outperform a deep ensemble while being significantly faster. NAPs are also shown to be able to predict performance metrics such as BERTScore (MT) or word error rate (ASR). For downstream tasks, such as data filtering and resource optimization, NAPs generate performance predictions that outperform predictive uncertainty while being highly inference efficient.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 75 references that are in the map. First posted on arXiv 9 May 2023.
Builds on
46 of 75 references.
References · 46
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
34 more
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
- 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2015Sequence Level Training with Recurrent Neural Networkscited by 568
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
- 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
- 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
- 2020Fast Differentiable Sorting and Rankingcited by 196
- 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
- 2019Ensemble Distribution Distillationcited by 158
- 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
- 2021Improving black-box optimization in VAE latent space using decoder uncertaintycited by 51
- 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
- 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
- 2022Augmenting Softmax Information for Selective Classification with Out-of-Distribution Datacited by 27
- 2021Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targetscited by 19
- 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 15
- 2023DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentationcited by 15
- 2018Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networkscited by 12
- 2022Self-Distribution Distillation: Efficient Uncertainty Estimationcited by 11
- 2020Ensemble Distillation Approaches for Grammatical Error Correctioncited by 8
- 2019Predicting word error rate for reverberant speechcited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.