mlsanityData to 14 Oct 2026

Matryoshka Language Model Suites

arXiv 2608.09703 ↗First posted 10 Aug 2026cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking sub-models of increasing size into a single nested architecture trained end-to-end. This Matryoshka training framework reduces the total parameter count of the suite, enables low-cost distillation from the largest to all smaller sub-models at every training step, and is well-suited for speculative decoding as the draft model is contained within the verifier. We validate our approach by training a Matryoshka suite comprising 500M, 1.5B, and 3B sub-models. Our suite is on par with independently trained baselines on benchmark performance and validation and out-of-domain perplexities, while using 36% less training compute and improving the throughput of speculative decoding by 14-26%. We also ablate key architectural choices, offering guidance for building strong Matryoshka LM suites.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 41 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

34 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
17 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  4. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  5. 2022Confident Adaptive Language Modelingcited by 310
  6. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  7. 2018Slimmable Neural Networkscited by 305
  8. 2022Matryoshka Representation Learningcited by 282
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  10. 2019Depth-Adaptive Transformercited by 236
  11. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  12. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  13. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 85
  14. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 45
  15. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 38
  16. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  17. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
  4. 2025MatryoshkaThinking: Recursive Test-Time Scaling Enables Efficient Reasoning
  5. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  6. 2024Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding