Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone language model family grown in four stages from a small dense seed, through a 5B and a 9B mixture of experts, to a 120B model with 460 routed experts under top-12 routing. Each larger model is grown from the trained weights of the smaller one; active parameters rise monotonically from 1.78B at the dense seed to 5.93B at 120B (about 5% of the 118.67B stored). The full lineage runs on single nodes, the larger stages at 8K context, reaching a released training loss of 1.78 at 120B scale. This is a systems and experience report. It is organized around three disciplines. Reversibility: a reversible recurrence stack reconstructs activations in the backward pass instead of storing them, holding activation memory flat as the model grows. State-preserving growth: each expansion (dense to MoE, shallow to deep, few experts to many) is given as a reproducible principle paired with the failure that results from getting it wrong; several failures are silent. Single-node economics: the 120B trains through TQP, a strategy of quantized base expert weights and trained low-rank adapters that carries optimizer state on 2.26B adapter parameters rather than 100B+ resident in routed experts, cutting expert-path optimizer state by a factor of ~45. What is new is the integration of known primitives, not any primitive in isolation: one grown lineage running end to end on a single node, documented at practitioner level, with per-domain held-out loss as evidence that targeted capabilities (multilingual Indic competence, code) were learned by construction. Model family, tokenizer, and training code are released.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 38 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
36 of 38 references.
Closest ancestors · 6
Same direction, newest first
- 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts11 Aug
- 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling24 Jul
- 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization26 Feb
- 2024Upcycling Large Language Models into Mixture of Experts10 Oct
- 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan
Other references · 30
- 2023GPT-4 Technical Reportcited by 18,132
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
- 2020Query-Key Normalization for Transformerscited by 266
- 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
- 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
18 more
- 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
- 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
- 2024Hyper-Connectionscited by 81
- 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
- 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
- 2024Weight decay induces low-rank attention layerscited by 27
- 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
- 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
- 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
- 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
- 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
- 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
- 2026The Depth Delusion: Why Transformers Should Be Wider, Not Deepercited by 2
- 2026BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_basecited by 2
- 2025Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Modelscited by 1
- 2025Reversing Large Language Models for Efficient Training and Fine-Tuningcited by 1
- 2026Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks
- 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
- 2026Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
- 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
- 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
- 2026EMO: Frustratingly Easy Progressive Training of Extendable MoE
- 2026Scaling Laws for Looped Mixture of Experts
- 2026Sparse Layers are Critical to Scaling Looped Language Models