An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
We report an exploratory, single-seed ablation of TALH (Adaptive Latent Hybrid), a decoder-only language model with parallel Multi-head Latent Attention (MLA) and a custom recurrent state-space (SSM) branch. Five variants, spanning 117--217M estimated active parameters per token, are trained from scratch on a FineWeb sample for the same number of optimisation steps and tokens. In this specific setup, removing the SSM branch gives the largest degradation in validation perplexity (MLA-only PPL 315), whereas removing MLA has a much smaller effect (SSM-only PPL 239). A dense-FFN hybrid obtains PPL 231, compared with 240 for the tested top-2 ternary-MoE hybrid, while using 3.87 GB less peak training memory. We also preserve a preliminary Apple M3 timing observation: among the five unoptimised implementations, MLA-only has the flattest measured time-to-first-token curve from 512 to 2,048 prompt tokens, although the dense Transformer is much faster in absolute terms. Because the runs are single-seed, parameter counts are unmatched, the evaluation stream may overlap the training source, and raw repeated timing records are unavailable, these results support implementation-specific hypotheses rather than general conclusions about MLA, SSMs, or mixture-of-experts models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 13 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
12 of 13 references.
Closest ancestors · 2
Same direction, newest first
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
- 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
Other references · 10
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
- 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
- 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
- 2025BitNet b1.58 2B4T Technical Reportcited by 30
- 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
- 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
- 2025Latent Multi-Head Attention for Small Language Models
- 2026LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
- 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
- 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
- 2026FLARE: Diffusion for Hybrid Language Model
- 2026dQwen3.5: Hybrid-Attention Diffusion Language Models
- 2026Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing