mlsanityData to 14 Oct 2026

Mimetic Initialization of Self-Attention Layers

arXiv 2305.09828 ↗First posted 16 May 2023cs.CV cs.AI cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

It is notoriously difficult to train Transformers on small datasets; typically, large pre-trained models are instead used as the starting point. We explore the weights of such pre-trained Transformers (particularly for vision) to attempt to find reasons for this discrepancy. Surprisingly, we find that simply initializing the weights of self-attention layers so that they "look" more like their pre-trained counterparts allows us to train vanilla Transformers faster and to higher final accuracies, particularly on vision tasks such as CIFAR-10 and ImageNet classification, where we see gains in accuracy of over 5% and 4%, respectively. Our initialization scheme is closed form, learning-free, and very simple: we set the product of the query and key weights to be approximately the identity, and the product of the value and projection weights to approximately the negative identity. As this mimics the patterns we saw in pre-trained Transformers, we call the technique "mimetic initialization".

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 19 of its 21 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

19 of 21 references.

References · 19

  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  5. 2021Going deeper with Image Transformerscited by 764
  6. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  7. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  8. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  9. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  10. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  11. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  12. 2022Patches Are All You Need?cited by 201
7 more
  1. 2021Vision Transformer for Small-Size Datasetscited by 122
  2. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  3. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  4. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  5. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 35
  6. 2022Training Vision Transformers with Only 2040 Imagescited by 23
  7. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 10; 2026: 18.

54 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2024Kolmogorov-Arnold Transformercited by 56
  3. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
  4. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
  5. 2023Initializing Models with Larger Onescited by 39
  6. 2023Simplifying Transformer Blockscited by 37
  7. 2023When can transformers reason with abstract symbols?cited by 24
  8. 2023LEMON: Lossless model expansioncited by 23
  9. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  10. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  11. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 18
  12. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
38 more
  1. 2024WAVE: Weight Templates for Adaptive Initialization of Variable-sized Modelscited by 17
  2. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 17
  3. 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
  4. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  5. 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 12
  6. 2024When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Modelscited by 12
  7. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  8. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingcited by 9
  9. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  10. 2025Cutting the Skip: Training Residual-Free Transformerscited by 8
  11. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  12. 2025In-context denoising with one-layer transformers: connections between attention and associative memory retrievalcited by 7
  13. 2025Structured Initialization for Vision Transformerscited by 6
  14. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  15. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  16. 2024Convolutional Initialization for Data-Efficient Vision Transformerscited by 4
  17. 2024Structured Initialization for Attention in Vision Transformerscited by 4
  18. 2024Reasoning in Large Language Models: A Geometric Perspectivecited by 4
  19. 2024Long-term Pre-training for Temporal Action Detection with Transformerscited by 3
  20. 2025FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformercited by 3
  21. 2026Attention Transfer Is Not Universally Effective for Vision Transformerscited by 3
  22. 2024Discrete Cosine Transform Based Decorrelated Attention for Vision Transformerscited by 2
  23. 2026Mimetic Initialization of MLPscited by 2
  24. 2026A Unified Framework for Knowledge Transfer in Bidirectional Model Scalingcited by 2
  25. 2023Robust 6DoF Pose Estimation Against Depth Noise and a Comprehensive Evaluation on a Mobile Datasetcited by 1
  26. 2024Freqformer: Frequency-Domain Transformer for 3-D Reconstruction and Quantification of Human Retinal Vasculaturecited by 1
  27. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  28. 2026Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Modelscited by 1
  29. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 1
  30. 2026Making Transformers Logicalcited by 1
  31. 2025Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classificationcited by 0
  32. 2026Self-Supervised Weight Templates for Scalable Vision Model Initializationcited by 0
  33. 2026Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transformcited by 0
  34. 2026Constraint-based Pre-training: From Structured Constraints to Scalable Model Initializationcited by 0
  35. 2026Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRIcited by 0
  36. 2026One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Contextcited by 0
  37. 2026Conditioned Initialization for Attentioncited by 0
  38. 2026Copy the Same, Distill the Difference: Initializing Linear Vision Transformerscited by 0

50 most cited of 54 All 54 (JSON)