mlsanityData to 7 Oct 2026

Training-Free Looped Transformers

arXiv 2605.23872 ↗First posted 22 May 2026cs.LG cs.NA math.NAcited by 12

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm transformer block as a forward Euler step on an ODE, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our method improves Qwen3-4B-Instruct by +2.64 pp on MMLU-Pro, Qwen3-30B-A3B-Instruct by +1.14 pp on CommonsenseQA, and Moonlight-16B-A3B-Instruct by +1.20 pp on OpenBookQA.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 88 of its 102 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

88 of 102 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026φ-Balancing for Mixture-of-Experts Training14 May
  2. 2024MoEUT: Mixture-of-Experts Universal Transformers25 May

Other references · 86

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  11. 2018Universal Transformerscited by 613
  12. 2019Deep Equilibrium Modelscited by 603
  13. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  14. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  15. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  16. 2025Muon is Scalable for LLM Trainingcited by 381
  17. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  18. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  19. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  20. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  21. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  22. 2023Looped Transformers as Programmable Computerscited by 224
  23. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  24. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  25. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  26. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  27. 2021PonderNet: Learning to Pondercited by 156
  28. 2020Multiscale Deep Equilibrium Modelscited by 149
  29. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  30. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  31. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  32. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  33. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  34. 2024Looped Transformers for Length Generalizationcited by 91
  35. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  36. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  37. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  38. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026On the Residual Scaling of Looped Transformers: Stability and Transferabilitycited by 7
  2. 2026Loop the Loopies!cited by 4
  3. 2026AGI Maze as a Benchmark Framework for World-Modeling Agentscited by 2
  4. 2026Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Modelscited by 1
  5. 2026Penelope: Localized Latent Recurrence for Efficient Structured Reasoningcited by 0
  6. 2026Write Back the Delta: Revisiting the Same Tokens with Fresh Representationscited by 0
  7. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 0
  8. 2026Scheduling Recursive Reasoning in Looped Transformerscited by 0
  9. 2026Looped Transformers as Optimizerscited by 0
  10. 2026Decoding Looped Transformers Better for (Almost) Freecited by 0
  11. 2026Enabling Dynamic Computation in Looped LMscited by 0
  12. 2026Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
  2. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?
  3. 2026Step-resolved data attribution for looped transformers
  4. 2026Simply Stabilizing the Loop via Fully Looped Transformer
  5. 2026Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
  6. 2026One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models
  7. 2026LoopMTP: A looped transformer guided by latent multi-token prediction
  8. 2026Stability and Generalization in Looped Transformers