mlsanityData to 14 Oct 2026

Pay Attention to Small Weights

arXiv 2506.21374 ↗First posted 26 Jun 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free -- the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

35 of 49 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
23 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 675
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  6. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  7. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 334
  8. 2024LoRA Learns Less and Forgets Lesscited by 328
  9. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 201
  10. 2023Efficient Large Language Models: A Surveycited by 186
  11. 2024Mixture of LoRA Expertscited by 181
  12. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  13. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  14. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  15. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  16. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  17. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  18. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  19. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  20. 2015The Singular Value Decomposition, Applications and Beyondcited by 11
  21. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  22. 2025Sparsity May Be All You Need: Sparse Random Parameter Adaptationcited by 5
  23. 2024Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 4
  2. 2026Learning in the Null Space: Small Singular Values for Continual Learningcited by 1
  3. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0