# The Universal Weight Subspace Hypothesis

arXiv: https://arxiv.org/abs/2512.05117 · HTML: https://mlsanity.com/paper/2512.05117/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel](https://mlsanity.com/paper/1810.05369/) (2018)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS](https://mlsanity.com/paper/2009.10683/) (2020)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead](https://mlsanity.com/paper/2407.00066/) (2024)
- [Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks](https://mlsanity.com/paper/2104.03093/) (2021)
- [The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold](https://mlsanity.com/paper/2305.01604/) (2023)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [An Equivalence of Fully Connected Layer and Convolutional Layer](https://mlsanity.com/paper/1712.01252/) (2017)
- [SETOL: A Semi-Empirical Theory of (Deep) Learning](https://mlsanity.com/paper/2507.17912/) (2025)
- [EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference](https://mlsanity.com/paper/2502.04700/) (2025)
- [Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation](https://mlsanity.com/paper/2401.10848/) (2024)

## Built on by (most cited first, 24 of 24)

- [Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization](https://mlsanity.com/paper/2602.17155/) (2026)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability](https://mlsanity.com/paper/2606.06333/) (2026)
- [From Adversarial Poetry to Adversarial Tales: An Interpretability Research Agenda](https://mlsanity.com/paper/2601.08837/) (2025)
- [Shared LoRA Subspaces for almost Strict Continual Learning](https://mlsanity.com/paper/2602.06043/) (2026)
- [On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR](https://mlsanity.com/paper/2605.06523/) (2026)
- [Spectral phase transitions and trainability in neural network learning dynamics](https://mlsanity.com/paper/2606.28486/) (2026)
- [Beyond Weight Adaptation: Feature-Space Domain Injection for Cross-Modal Ship Re-Identification](https://mlsanity.com/paper/2512.20892/) (2025)
- [LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules](https://mlsanity.com/paper/2602.10993/) (2026)
- [Black-box model classification under the discriminative factorization](https://mlsanity.com/paper/2605.07878/) (2026)
- [FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning](https://mlsanity.com/paper/2605.09355/) (2026)
- [Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered](https://mlsanity.com/paper/2605.15622/) (2026)
- [DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions](https://mlsanity.com/paper/2607.20469/) (2026)
- [Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws](https://mlsanity.com/paper/2608.13335/) (2026)
- [SHARe-KAN: Post-Training Vector Quantization for Cache-Resident KAN Inference](https://mlsanity.com/paper/2512.15742/) (2025)
- [Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity](https://mlsanity.com/paper/2512.20291/) (2025)
- [Universal Latent Homeomorphic Manifolds: A Framework for Cross-Domain Representation Unification](https://mlsanity.com/paper/2601.09025/) (2026)
- [ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism](https://mlsanity.com/paper/2604.11947/) (2026)
- [A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws](https://mlsanity.com/paper/2604.24037/) (2026)
- [Metaphor Is Not All Attention Needs](https://mlsanity.com/paper/2605.12128/) (2026)
- [The Cross-Architecture Substrate: A Domain-Transcendent, Calibration-Surviving Geometric Invariant of Modern Vision Encoders](https://mlsanity.com/paper/2606.07882/) (2026)
- [LoRSA: Toward Generalizable Parameter-Efficient Fine-Tuning for Biomedical Downstream Tasks](https://mlsanity.com/paper/2608.07749/) (2026)
- [Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space](https://mlsanity.com/paper/2608.07786/) (2026)
- [Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities](https://mlsanity.com/paper/2609.24209/) (2026)

## Abstract (arXiv)

We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1200 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.