# Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades

arXiv: https://arxiv.org/abs/2501.10661 · HTML: https://mlsanity.com/paper/2501.10661/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2025.

## Built on by (most cited first, 8 of 8)

- [Co-Reinforcement Learning for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2505.17534/) (2025)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [Why Do More Experts Fail? A Theoretical Analysis of Model Merging](https://mlsanity.com/paper/2505.21226/) (2025)
- [Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations](https://mlsanity.com/paper/2504.00851/) (2025)
- [NAN: A Training-Free Solution to Coefficient Estimation in Model Merging](https://mlsanity.com/paper/2505.16148/) (2025)
- [ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training](https://mlsanity.com/paper/2604.27844/) (2026)
- [SBVR: Summation of BitVector Representation for Efficient LLM Quantization](https://mlsanity.com/paper/2509.18172/) (2025)
- [AMS-QUANT: Adaptive Mantissa Sharing for Floating-point Quantization](https://mlsanity.com/paper/2510.16045/) (2025)

## Abstract (arXiv)

This paper presents a pioneering exploration of the mechanisms underlying large foundation models' (LFMs) weights, aiming to simplify AI research. Through extensive observation and analysis on prevailing LFMs, we find that regardless of initialization strategies, their weights predominantly follow a Gaussian distribution, with occasional sharp, inverted T-shaped, or linear patterns. We further discover that the weights share the i.i.d. properties of Gaussian noise, and explore their direct relationship. We find that transformation weights can be derived from Gaussian noise, and they primarily serve to increase the standard deviation of pre-trained weights, with their standard deviation growing with layer depth. In other words, transformation weights broaden the acceptable deviation from the optimal weights, facilitating adaptation to downstream tasks. Building upon the above conclusions, we thoroughly discussed the nature of optimal weights, ultimately concluding that they should exhibit zero-mean, symmetry, and sparsity, with the sparse values being a truncated Gaussian distribution and a few outliers. Our experiments in LFM adaptation and editing demonstrate the effectiveness of these insights. We hope these findings can provide a foundational understanding to pave the way for future advancements in the LFM community.