mlsanityData to 7 Oct 2026

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation

arXiv 2502.03029 ↗First posted 5 Feb 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

47 of 63 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
35 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  12. 2023Instruction Tuning with GPT-4cited by 676
  13. 2024Large Language Models: A Surveycited by 506
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  16. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  17. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  18. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  19. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  20. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  21. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  22. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  23. 2019Convergence Rates for Gaussian Mixtures of Expertscited by 39
  24. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  25. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  26. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  27. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  28. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  29. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
  30. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  31. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  32. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  33. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  34. 2024Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Expertscited by 5
  35. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attentioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learningcited by 2
  2. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  3. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  4. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
  3. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
  4. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
  6. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
  7. 2026When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
  8. 2024MoH: Multi-Head Attention as Mixture-of-Head Attention