mlsanityData to 14 Oct 2026

Loop the Loopies!

arXiv 2607.16051 ↗First posted 17 Jul 2026cs.CL cs.AIcited by 8

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 125 of its 147 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

125 of 147 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Training-Free Looped Transformers22 May
  2. 2024MoEUT: Mixture-of-Experts Universal Transformers25 May
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 122

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
38 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2018Universal Transformerscited by 647
  11. 2019Deep Equilibrium Modelscited by 615
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  13. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  14. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  15. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  16. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  17. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  18. 2023Looped Transformers as Programmable Computerscited by 237
  19. 2019Depth-Adaptive Transformercited by 236
  20. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  21. 2022Recurrent Memory Transformercited by 203
  22. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
  23. 2025Hierarchical Reasoning Modelcited by 177
  24. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  25. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  26. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  27. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  28. 2020Multiscale Deep Equilibrium Modelscited by 151
  29. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  30. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  31. 2015Neural GPUs Learn Algorithmscited by 121
  32. 2022Block-Recurrent Transformerscited by 110
  33. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  34. 2024Looped Transformers for Length Generalizationcited by 99
  35. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
  36. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  37. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  38. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  2. 2026Depth through recurrence: Looped transformers for flow-matching TTScited by 1
  3. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1
  4. 2026Closing the Loop: Practical Training Recipes for Looped Language Modelscited by 1
  5. 2026Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Expertscited by 0
  6. 2026ALoDLM: Adaptively Looped Diffusion Language Modelscited by 0
  7. 2026Towards Looped Models Done Right, Part II: Rethinking at Fixed Pointscited by 0
  8. 2026Enabling Dynamic Computation in Looped LMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Laws for Looped Mixture of Experts
  2. 2026LoopMTP: A looped transformer guided by latent multi-token prediction
  3. 2026How to Loop MoE: Flatten the Experts, Untie the Attention
  4. 2026Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
  5. 2026Step-resolved data attribution for looped transformers
  6. 2026LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
  7. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  8. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers