mlsanityData to 14 Oct 2026

Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M

arXiv 2609.14715 ↗First posted 13 Sep 2026cs.AI cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 15 of its 17 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

15 of 17 references.

References · 15

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2019Root Mean Square Layer Normalizationcited by 1,480
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
3 more
  1. 2020Query-Key Normalization for Transformerscited by 280
  2. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  3. 2024Value Residual Learningcited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
  2. 2026Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
  3. 2026Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
  4. 2025Efficient Pretraining Length Scaling
  5. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  6. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  7. 2026Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
  8. 2026Super Apriel: One Checkpoint, Many Speeds