mlsanityData to 14 Oct 2026

Exploring and Reshaping the Weight Distribution in LLM

arXiv 2509.00046 ↗First posted 24 Aug 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The performance of Large Language Models is influenced by their characteristics such as architecture, model sizes, decoding methods and so on. Due to differences in structure or function, the weights in different layers of large models have varying distributions. This paper explores the correlations between different types of layers in terms of weights distribution and studies the potential impact of these correlations on LoRA training effectiveness. Firstly, the study reveals that in the model the cosine distances between weights of different layers manifest power-law distribution. We extract Query-projection, down-projection and other weight matrices from the self-attention layers and MLP layers, calculate the singular values of the matrices using singular value decomposition, and organize a certain number of singular values into matrices according to projection's type. By analyzing the probability distribution of the cosine distances between these matrices, it is found that the cosine distances values between them have distinct power-law distribution characteristics. Secondly, based on the results of distance calculations and analysis across different layers of model, a qualitative method is proposed to describe the distribution characteristics of different models. Next, to construct weights that align with the distribution characteristics, a data generator is designed using a combination of Gaussian process and Pareto distribution functions. The generator is used to simulate the generation of data that aligns with specific distribution characteristics. Finally, based on the aforementioned distribution characteristics and data generation method, the weights in LoRA initialization are reshaped for training. Experimental results indicate that, without altering the model structure or training process, this method achieves a certain improvement in the performance of LoRA training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

17 of 23 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  5. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  7. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  8. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  9. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  10. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  11. 2024A Survey on LoRA of Large Language Modelscited by 67
  12. 2024The Super Weight in Large Language Modelscited by 52
5 more
  1. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  2. 2020Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruningcited by 30
  3. 2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Modelscited by 15
  4. 2024Importance Weighting Can Help Large Language Models Self-Improvecited by 12
  5. 2023Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?cited by 8

Built on by

Not cited yet.