mlsanityData to 7 Oct 2026

DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging

arXiv 2402.02622 ↗First posted 4 Feb 2024cs.CL cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations -- we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

26 of 37 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
14 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  7. 2015Highway Networkscited by 257
  8. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  9. 2014Convolutional Neural Networks at Constrained Time Costcited by 81
  10. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  11. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  12. 2023Simplifying Transformer Blockscited by 37
  13. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  14. 2022Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classificationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 34.

48 papers.

Sort:

By area:

  1. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
  2. 2026Attention Residualscited by 57
  3. 2024Transformer Layers as Painterscited by 50
  4. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  5. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  6. 2024GPT or BERT: why not both?cited by 26
  7. 2024Value Residual Learningcited by 25
  8. 2026Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deservescited by 11
  9. 2026Hyperloop Transformerscited by 11
  10. 2025DeepCrossAttention: Supercharging Transformer Residual Connectionscited by 7
  11. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  12. 2026Deep Delta Learningcited by 7
36 more
  1. 2026Delta Attention Residualscited by 7
  2. 2025Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlookcited by 6
  3. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
  4. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  5. 2025You Do Not Fully Utilize Transformer's Representation Capacitycited by 4
  6. 2025Auto-Compressing Networkscited by 4
  7. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  8. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 4
  9. 2026Mixture-of-Depths Attentioncited by 4
  10. 2026ZAYA1-8B Technical Reportcited by 4
  11. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 3
  12. 2026Depth-Attention: Cross-Layer Value Mixing for Language Modelscited by 3
  13. 2026xHC: Expanded Hyper-Connectionscited by 3
  14. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmissioncited by 2
  15. 2026Residual Stream Duality in Modern Transformer Architecturescited by 2
  16. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformerscited by 2
  17. 2026ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generationcited by 2
  18. 2026Multi-Head Attention Residualscited by 2
  19. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  20. 2026Improving LLM Final Representations with Inter-Layer Geometrycited by 1
  21. 2026Transformers with Selective Access to Early Representationscited by 1
  22. 2026Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floorcited by 1
  23. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  24. 2026Low-Rank Attention Residualscited by 1
  25. 2026LayerRoute: Action-Conditioned Mixture-of-Layers Routing for Vision-Language-Action Policiescited by 1
  26. 2026YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inferencecited by 0
  27. 2026HAARES Half-Split Residual Basis Routing for Deep Transformerscited by 0
  28. 2026ZONOS2 Technical Reportcited by 0
  29. 2026Dual Attention Residualscited by 0
  30. 2026Dynamic Parameterization Is Not Dynamic Inferencecited by 0
  31. 2026Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depthcited by 0
  32. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  33. 2026NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Predictioncited by 0
  34. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  35. 2026DepthBench: Measuring How Residual Connections Enable More Computational Depthcited by 0
  36. 2026T-Router: Learning Thalamic Routing for Reasoning with Parameter-Efficient Reinforcement Learningcited by 0