mlsanityData to 7 Oct 2026

The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

arXiv 2202.05798 ↗First posted 11 Feb 2022cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the entire training experience. While this has been technically known since the 1960s, no prior work has effectively studied the operations of NNs in such a form, presumably due to prohibitive time and space complexities and impractical model sizes, all of them growing linearly with the number of training patterns which may get very large. However, this dual formulation offers a possibility of directly visualising how an NN makes use of training patterns at test time, by examining the corresponding attention weights. We conduct experiments on small scale supervised image classification tasks in single-task, multi-task, and continual learning settings, as well as language modelling, and discuss potentials and limits of this view for better understanding and interpreting how NNs exploit training patterns. Our code is public.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

22 of 36 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
10 more
  1. 2014Neural Turing Machinescited by 630
  2. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  3. 2021Random Feature Attentioncited by 350
  4. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  5. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  6. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  7. 2019Language Modeling with Deep Transformerscited by 96
  8. 2019Augmenting Self-attention with Persistent Memorycited by 92
  9. 2021Learning in High Dimension Always Amounts to Extrapolationcited by 62
  10. 2020Every Model Learned by Gradient Descent Is Approximately a Kernel Machinecited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 14; 2024: 15; 2025: 4; 2026: 5.

45 papers.

Sort:

By area:

  1. 2022A Survey on In-context Learningcited by 717
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  5. 2024Titans: Learning to Memorize at Test Timecited by 303
  6. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  7. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  8. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  9. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  10. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  11. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  12. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
33 more
  1. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  2. 2025Key-value memory in the braincited by 22
  3. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  4. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  5. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  6. 2022Discrete Key-Value Bottleneckcited by 18
  7. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15
  8. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15
  9. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  10. 2022BayesPCN: A Continually Learnable Predictive Coding Associative Memorycited by 12
  11. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  12. 2024Universal Link Predictor By In-Context Learning on Graphscited by 12
  13. 2023Sparse Distributed Memory is a Continual Learnercited by 11
  14. 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9
  15. 2023Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questionscited by 9
  16. 2024Focused Large Language Models are Stable Many-Shot Learnerscited by 8
  17. 2024A Theory for Token-Level Harmonization in Retrieval-Augmented Generationcited by 7
  18. 2023Metalearning Continual Learning Algorithmscited by 6
  19. 2024Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformerscited by 6
  20. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
  21. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  22. 2023A Meta-Learning Perspective on Transformers for Causal Language Modelingcited by 5
  23. 2024Locally Differentially Private In-Context Learningcited by 5
  24. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  25. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  26. 2023Causal Intersectionality and Dual Form of Gradient Descent for Multimodal Analysis: a Case Study on Hateful Memescited by 2
  27. 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networkscited by 1
  28. 2023Accelerating Neural Self-Improvement via Bootstrappingcited by 1
  29. 2023Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layercited by 1
  30. 2026Modular TTT: Rethinking Test-Time Training as Composable Modulescited by 1
  31. 2026IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Modelscited by 0
  32. 2026Understanding In-Context Multimodal Jailbreaks via Posterior Reweightingcited by 0
  33. 2026Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modelingcited by 0