mlsanityData to 14 Oct 2026

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

arXiv 2606.31796 ↗First posted 30 Jun 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Frontier language capability is usually bought with frontier compute; CHERRY shows a different trade. It is a sovereign Korean model family built on one principle: supervise the tokens that decide the answer, and let shared weights carry the rest. Under matched compute this exposes a sharp, reproducible dissociation---selected-token supervision preserves held-out discrimination yet collapses free generation, and a full-sequence anchor recovers only part of the gap. The same signal drives a heal-after-merge recurrent-representational-yield loop that collapses 48 layers to 6 unique blocks at near-dense parity (227M at loss 2.934 vs a 566M dense model at 2.926) and composes them by MoEE fusion (2.789)---a recurrent-compression direction independently pursued by concurrent frontier looped-MoE work, which we project (not yet measure) to frontier scale. It also installs metacognition from two-token supervision (200 held-out KO prompts/type, kappa>0.82, +/-6.9pp): self-correction 12->47% and jailbreak 23->4% at 97.6% loss-retention on 1.2B, with a pre-registered 1B->13.7B ablation localising the operand-binding limit to capacity (1B lookup vs 13.7B H-PRESERVE); and it specializes a 1.8B model to reported human-expert level on CyberMetric (75.0% vs a 30-expert average 72.24%). The released 1.8B tokenizer is 9.2% more Korean-efficient than Gemma-4; the from-scratch 12B adds a sovereign Korean tokenizer (vocab 131,037). On the government-operated K-AI Korean-LLM leaderboard the released 1.8B leads the HLE(Ko) column (0.123 vs 0.077; overall 51/78). At the frontier we fine-tune and serve an adapted 122B model on one 120GB accelerator (83GB measured peak). Provenance differs by member and we state it exactly; the architecture and recipe are ours throughout, and every claim is bound to a released curated measurement.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

31 of 48 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
19 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  7. 2018Universal Transformerscited by 647
  8. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  9. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  12. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  13. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  14. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  15. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  16. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  17. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  18. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  19. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Loop the Loopies!cited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
  2. 2026Compute Where it Counts: Self Optimizing Language Models
  3. 2023Just CHOP: Embarrassingly Simple LLM Compression
  4. 2024Compact Language Models via Pruning and Knowledge Distillation
  5. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  6. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
  7. 2026DiffuMask: Diffusion Language Model for Token-level Prompt Pruning
  8. 2026What Survives When You Compress a Recursive Reasoner for the Edge?