mlsanityData to 14 Oct 2026

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation

arXiv 2507.10524 ↗First posted 14 Jul 2025cs.CL cs.LGcited by 133

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation, leaving open the question of how to attain both simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer. MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking by dynamically assigning different recursion depths to individual tokens. This allows MoR to focus quadratic attention computation only among tokens still active at a given recursion depth, further improving memory access efficiency by selectively caching only their key-value pairs. Beyond these core mechanisms, we also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to further decrease memory footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms a new Pareto frontier: at equal training FLOPs and smaller model sizes, it significantly lowers validation perplexity and improves few-shot accuracy, while delivering higher throughput compared with vanilla and existing recursive baselines. These gains demonstrate that MoR is an effective path towards large-model quality without incurring large-model cost.

Key facts. Cited by 133 papers in the map (data through 14 Oct 2026). Builds on 99 of its 112 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

99 of 112 references.

References · 99

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2024Mixtral of Expertscited by 1,823
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  13. 2016Deep Networks with Stochastic Depthcited by 979
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  15. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  16. 2020Efficient Transformers: A Surveycited by 790
  17. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  19. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  20. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  21. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  22. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  23. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  24. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  25. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  26. 2022Efficiently Scaling Transformer Inferencecited by 539
  27. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  28. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  29. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  30. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  31. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  32. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  33. 2022Confident Adaptive Language Modelingcited by 310
  34. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  35. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  36. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  37. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  38. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 108.

133 papers.

Sort:

By area:

  1. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  2. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  3. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  4. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  5. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 33
  6. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  7. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  8. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 28
  9. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  10. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 24
  11. 2026Generative Recursive Reasoningcited by 20
  12. 2026Probabilistic Tiny Recursive Modelcited by 19
38 more
  1. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 18
  2. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  3. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  4. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 16
  5. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 15
  6. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 15
  7. 2026Training-Free Looped Transformerscited by 15
  8. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 14
  9. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 14
  10. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  11. 2025Dr.LLM: Dynamic Layer Routing in LLMscited by 12
  12. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  13. 2026Hyperloop Transformerscited by 12
  14. 2026Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readoutscited by 12
  15. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 11
  16. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  17. 2026Sparse Layers are Critical to Scaling Looped Language Modelscited by 11
  18. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 11
  19. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 10
  20. 2026Recursive Multi-Agent Systemscited by 10
  21. 2026LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modelingcited by 10
  22. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 9
  23. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  24. 2026Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs?cited by 8
  25. 2026Loop the Loopies!cited by 8
  26. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  27. 2025On the Reasoning Abilities of Masked Diffusion Language Modelscited by 7
  28. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 7
  29. 2026LT2: Linear-Time Looped Transformerscited by 7
  30. 2025Route Experts by Sequence, not by Tokencited by 6
  31. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  32. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 6
  33. 2026Understanding Dynamic Compute Allocation in Recurrent Transformerscited by 6
  34. 2026LoopQ: Quantization for Looped Language Modelscited by 6
  35. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 6
  36. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  37. 2026Step-resolved data attribution for looped transformerscited by 5
  38. 2026Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoningcited by 5

50 most cited of 133 All 133 (JSON)