mlsanityData to 7 Oct 2026

Modeling Recurrence for Transformer

arXiv 1904.03092 ↗First posted 5 Apr 2019cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Recently, the Transformer model that is based solely on attention mechanisms, has advanced the state-of-the-art on various machine translation tasks. However, recent studies reveal that the lack of recurrence hinders its further improvement of translation capacity. In response to this problem, we propose to directly model recurrence for Transformer with an additional recurrence encoder. In addition to the standard recurrent neural network, we introduce a novel attentive recurrent network to leverage the strengths of both attention and recurrent networks. Experimental results on the widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness of the proposed approach. Our studies also reveal that the proposed model benefits from a short-cut that bridges the source and target sequences with a single recurrent layer, which outperforms its deep counterpart.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 12; 2023: 3; 2024: 2; 2025: 2.

26 papers.

Sort:

By area:

  1. 2021Random Feature Attentioncited by 350
  2. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  3. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  4. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  5. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  6. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  7. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  8. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  9. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  10. 2023Learning to Solve Constraint Satisfaction Problems with Recurrent Transformercited by 21
  11. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  12. 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
14 more
  1. 2022On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translationcited by 15
  2. 2021Staircase Attention for Recurrent Processing of Sequencescited by 13
  3. 2024Neuro-symbolic Learning Yielding Logical Constraintscited by 12
  4. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  5. 2022BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representationcited by 8
  6. 2022Implicit N-grams Induced by Recurrencecited by 4
  7. 2022Simple Recurrence Improves Masked Language Modelscited by 4
  8. 2023Understanding Parameter Sharing in Transformerscited by 3
  9. 2022Exploring the sequence length bottleneck in the Transformer for Image Captioningcited by 2
  10. 2025Integrating Vision and Location with Transformers: A Multimodal Deep Learning Framework for Medical Wound Analysiscited by 2
  11. 2023Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translationcited by 1
  12. 2025Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architecturescited by 1
  13. 2021To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graphcited by 0
  14. 2022Hysteretic Behavior Simulation Based on Pyramid Neural Network:Principle, Network Architecture, Case Study and Explanationcited by 0