mlsanityData to 7 Oct 2026

Parallel Attention Forcing for Machine Translation

arXiv 2211.03237 ↗First posted 6 Nov 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Attention-based autoregressive models have achieved state-of-the-art performance in various sequence-to-sequence tasks, including Text-To-Speech (TTS) and Neural Machine Translation (NMT), but can be difficult to train. The standard training approach, teacher forcing, guides a model with the reference back-history. During inference, the generated back-history must be used. This mismatch limits the evaluation performance. Attention forcing has been introduced to address the mismatch, guiding the model with the generated back-history and reference attention. While successful in tasks with continuous outputs like TTS, attention forcing faces additional challenges in tasks with discrete outputs like NMT. This paper introduces the two extensions of attention forcing to tackle these challenges. (1) Scheduled attention forcing automatically turns attention forcing on and off, which is essential for tasks with discrete outputs. (2) Parallel attention forcing makes training parallel, and is applicable to Transformer-based models. The experiments show that the proposed approaches improve the performance of models based on RNNs and Transformers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 6 Nov 2022.

Builds on

25 of 37 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  10. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  11. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
13 more
  1. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  2. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  3. 2018Scaling Neural Machine Translationcited by 204
  4. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  5. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  6. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  7. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  8. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  9. 2017Adversarial Neural Machine Translationcited by 21
  10. 2019Parallel Scheduled Samplingcited by 17
  11. 2019A New GAN-based End-to-End TTS Training Algorithmcited by 13
  12. 2019Attention Forcing for Sequence-to-sequence Model Trainingcited by 4
  13. 2021Attention Forcing for Machine Translationcited by 3

Built on by

Not cited yet.