mlsanityData to 7 Oct 2026

Sharing Attention Weights for Fast Transformer

arXiv 1906.11024 ↗First posted 26 Jun 2019cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Recently, the Transformer machine translation system has shown strong results by stacking attention layers on both the source and target-language sides. But the inference of this model is slow due to the heavy use of dot-product attention in auto-regressive decoding. In this paper we speed up Transformer via a fast and lightweight attention model. More specifically, we share attention weights in adjacent layers and enable the efficient re-use of hidden states in a vertical manner. Moreover, the sharing policy can be jointly learned with the MT model. We test our approach on ten WMT and NIST OpenMT tasks. Experimental results show that it yields an average of 1.3X speed-up (with almost no decrease in BLEU) on top of a state-of-the-art implementation that has already adopted a cache for fast inference. Also, our approach obtains a 1.8X speed-up when it works with the textscAan model. This is even 16 times faster than the baseline with no use of the attention cache.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 3; 2023: 8; 2024: 8; 2025: 9; 2026: 1.

34 papers.

Sort:

By area:

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  3. 2023Transformers in Speech Processing: A Surveycited by 38
  4. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  5. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  6. 2025Tensor Product Attention Is All You Needcited by 27
  7. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  8. 2021LazyFormer: Self Attention with Lazy Updatecited by 16
  9. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  10. 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
  11. 2021OH-Former: Omni-Relational High-Order Transformer for Person Re-Identificationcited by 10
  12. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
22 more
  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
  2. 2022ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioningcited by 6
  3. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  4. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  5. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  6. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  7. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  8. 2021Bag of Tricks for Optimizing Transformer Efficiencycited by 3
  9. 2022Hybrid-Regressive Neural Machine Translationcited by 3
  10. 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
  11. 2023RecycleGPT: An Autoregressive Language Model with Recyclable Modulecited by 3
  12. 2024PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecastingcited by 2
  13. 2025Deconstructing Attention: Investigating Design Principles for Effective Language Modelingcited by 2
  14. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  15. 2022PGX: A Multi-level GNN Explanation Framework Based on Separate Knowledge Distillation Processescited by 1
  16. 2023Only 5% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translationcited by 1
  17. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptationcited by 1
  18. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  19. 2025Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inferencecited by 0
  20. 2025ACE: A Cardinality Estimator for Set-Valued Queriescited by 0
  21. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
  22. 2026Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Modelscited by 0