mlsanityData to 14 Oct 2026

Value Residual Learning

arXiv 2410.17897 ↗First posted 23 Oct 2024cs.CLcited by 27

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers. This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals. And a variant is SVFormer, where all layers share the first layer's value embedding. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11% fewer model parameters and 20.3% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 41 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

33 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
19 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2016Deep Networks with Stochastic Depthcited by 979
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  13. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  14. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  15. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  16. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  17. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 53
  18. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 32
  19. 2024Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 24.

27 papers.

Sort:

By area:

  1. 2026Attention Residualscited by 65
  2. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  3. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  4. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  5. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
  6. 2026GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generationcited by 5
  7. 2025You Do Not Fully Utilize Transformer's Representation Capacitycited by 4
  8. 2026More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activationscited by 3
  9. 2026Depth-Attention: Cross-Layer Value Mixing for Language Modelscited by 3
  10. 2026YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inferencecited by 2
  11. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  12. 2026Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Noveltycited by 2
15 more
  1. 2026Transformers with Selective Access to Early Representationscited by 1
  2. 2026Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memorycited by 1
  3. 2026SNLP: Layer-Parallel Inference via Structured Newton Correctionscited by 1
  4. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  5. 2026Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingcited by 1
  6. 2026Attention Projection Mixing with Exogenous Anchorscited by 0
  7. 2026Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformerscited by 0
  8. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  9. 2026One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthographycited by 0
  10. 2026NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inferencecited by 0
  11. 2026Scaling Laws, Tabular Data and Actuarial Ratemaking Modelscited by 0
  12. 2026Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110Mcited by 0
  13. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  14. 2026Bongard: Training Machine Intuitioncited by 0
  15. 2026The Row Normalization Puzzle in Muoncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
  2. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
  3. 2026Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation
  4. 2026Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
  5. 2026Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
  6. 2026ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions
  7. 2026The Quantization Benefits of Residual-Free Transformers
  8. 2026Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison