Reinforcement Learning with Large Action Spaces for Neural Machine Translation
Not on the map. Why?
Abstract (arXiv)
Applying Reinforcement learning (RL) following maximum likelihood estimation (MLE) pre-training is a versatile method for enhancing neural machine translation (NMT) performance. However, recent work has argued that the gains produced by RL for NMT are mostly due to promoting tokens that have already received a fairly high probability in pre-training. We hypothesize that the large action space is a main obstacle to RL's effectiveness in MT, and conduct two sets of experiments that lend support to our hypothesis. First, we find that reducing the size of the vocabulary improves RL's effectiveness. Second, we find that effectively reducing the dimension of the action space without changing the vocabulary also yields notable improvement as evaluated by BLEU, semantic similarity, and human evaluation. Indeed, by initializing the network's final fully connected layer (that maps the network's internal dimension to the vocabulary dimension), with a layer that generalizes over similar actions, we obtain a substantial improvement in RL performance: 1.5 BLEU points on average.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 6 Oct 2022.
Builds on
29 of 57 references.
References · 29
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
- 2015Sequence Level Training with Recurrent Neural Networkscited by 573
- 2017Convolutional Sequence to Sequence Learningcited by 552
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 462
- 2017Six Challenges for Neural Machine Translationcited by 335
- 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
- 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
17 more
- 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
- 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
- 2015Minimum Risk Training for Neural Machine Translationcited by 115
- 2018Language GANs Falling Shortcited by 97
- 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 88
- 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
- 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
- 2019Learning Action Representations for Reinforcement Learningcited by 75
- 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
- 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
- 2018Fix your classifier: the marginal value of training the last weight layercited by 43
- 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
- 2019The Natural Language of Actionscited by 28
- 2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
- 2018Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translationcited by 10
- 2019Unsupervised Neural Machine Translation with SMT as Posterior Regularizationcited by 8
- 2018A Stochastic Decoder for Neural Machine Translationcited by 5
Built on by
Citing papers per year: 2023: 2.
2 papers.