mlsanityData to 14 Oct 2026

Deep Delta Learning

arXiv 2601.00417 ↗First posted 1 Jan 2026cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Transformer residual streams are updated by addition. A sufficiently expressive residual block can represent content replacement, but the residual update itself has no operation that reads, compares, and replaces content. We introduce Deep Delta Learning (DDL), which applies the delta rule over network depth. Each layer reads the residual state along a learned direction, compares the readout with a learned target, and writes a gated rank-1 correction back along the same direction. A closed gate gives the identity map, and a unit gate overwrites the selected readout with the target. DDL works with the usual vector state or with an expanded state that stores several value channels, while attention and MLP blocks keep the original model width. We pretrain decoder-only models with approximately GPT-2 small and medium sizes on the FineWeb-Edu dataset. At both scales, every DDL variant has lower validation loss and higher average one-shot accuracy than the additive baseline, and the best expanded variants raise that average by 0.91 and 1.18 points.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

40 of 57 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
28 more
  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  3. 2018Invertible Residual Networkscited by 365
  4. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  6. 2015Highway Networkscited by 260
  7. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  8. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  9. 2024Hyper-Connectionscited by 84
  10. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  11. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 63
  12. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  13. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 53
  14. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  15. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 31
  16. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  17. 2023Cross-Layer Retrospective Retrieving via Layer Attentioncited by 22
  18. 2020Highway Transformer: Self-Gating Enhanced Self-Attentive Networkscited by 18
  19. 2023Alternating Updates for Efficient Transformerscited by 13
  20. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 13
  21. 2025Residual Matrix Transformers: Scaling the Size of the Residual Streamcited by 12
  22. 2022O-ViT: Orthogonal Vision Transformercited by 10
  23. 2020N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equationscited by 9
  24. 2024LAuReL: Learned Augmented Residual Layercited by 7
  25. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  26. 2024Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformationcited by 5
  27. 2024OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problemscited by 5
  28. 2025HOFT: Householder Orthogonal Fine-tuningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Delta Attention Residualscited by 7
  2. 2026DeepLoop: Depth Scaling for Looped Transformerscited by 7
  3. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuningcited by 3
  4. 2026Residual Stream Duality in Modern Transformer Architecturescited by 3
  5. 2026Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regimecited by 2
  6. 2026Multi-Head Attention Residualscited by 2
  7. 2026Geometric and Dynamic Scaling in Deep Transformerscited by 0
  8. 2026Ontology Neural Networks for Topologically Conditioned Constraint Satisfactioncited by 0
  9. 2026Normality Constraint Learning: Adapting Foundation Models for Time Series Anomaly Detectioncited by 0