mlsanityData to 7 Oct 2026

Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

arXiv 2512.23448 ↗First posted 29 Dec 2025cs.LGcited by 1

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 7 of its 8 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

7 of 8 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning11 Sep

Other references · 6

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixture-of-Subspaces in Low-Rank Adaptation
  2. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
  3. 2025Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
  4. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations
  5. 2026More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models
  6. 2026Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
  7. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts
  8. 2026FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts