Distributionally Robust Recurrent Decoders with Random Network Distillation
Not on the map. Why?
Abstract (arXiv)
Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to "shortcut learning": relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 25 Oct 2021.
Builds on
33 of 40 references.
References · 33
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2016Layer Normalizationcited by 5,579
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2019Invariant Risk Minimizationcited by 1,832
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
21 more
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
- 2018Exploration by Random Network Distillationcited by 966
- 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
- 2017Dynamic Routing Between Capsulescited by 689
- 2015Sequence Level Training with Recurrent Neural Networkscited by 568
- 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
- 2019Compositionality decomposed: how do neural networks generalise?cited by 264
- 2021Towards Causal Representation Learningcited by 207
- 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
- 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
- 2019Distributionally Robust Language Modelingcited by 142
- 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
- 2019Domain Robustness in Neural Machine Translationcited by 64
- 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
- 2020Compositional Generalization by Learning Analytical Expressionscited by 53
- 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
- 2018Accelerating Neural Transformer via an Average Attention Networkcited by 22
- 2019Findings of the First Shared Task on Machine Translation Robustnesscited by 17
- 2018Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sumcited by 4
- 2019A Lightweight Recurrent Network for Sequence Modelingcited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.