mlsanityData to 7 Oct 2026

SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion

arXiv 2602.11698 ↗First posted 12 Feb 2026cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative refinement and latent reasoning. However, early looped Transformers often underperform non-recursive baselines of equal compute. While recent literature has introduced more effective recursion mechanisms to mitigate this gap, existing architectures still operate at a fixed, full-token resolution, neglecting the potential efficiency of computing over compressed latent representations. In this paper, we propose SpiralFormer, a looped Transformer that executes recurrence under a multi-resolution recursion schedule. We provide probing evidence that multi-resolution recursion enables the model to learn hierarchical dependencies by inducing iteration-wise functional specialization across different scales. Empirically, SpiralFormer achieves better parameter and compute efficiency than both looped and non-looped baselines across model scales from 160M to 1.4B, establishing sequence resolution as a potential axis for scaling recursive architectures.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 58 of its 60 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

58 of 60 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
38 more
  1. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  2. 2018Universal Transformerscited by 613
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2019Depth-Adaptive Transformercited by 227
  6. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  7. 2023Looped Transformers as Programmable Computerscited by 224
  8. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  9. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  10. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  11. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  12. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  13. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  14. 2024Looped Transformers for Length Generalizationcited by 91
  15. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  16. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  17. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  18. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 72
  19. 2025A Survey on Latent Reasoningcited by 71
  20. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  21. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  22. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  23. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  24. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  25. 2023Sparse Universal Transformercited by 28
  26. 2021Sliced Recursive Transformercited by 26
  27. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  28. 2025Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Modelscited by 22
  29. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  30. 2024Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformercited by 16
  31. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 14
  32. 2025Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacecited by 14
  33. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  34. 2025Continuous Autoregressive Language Modelscited by 9
  35. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  36. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  37. 2025Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modelingcited by 7
  38. 2025AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modelingcited by 7

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Training-Free Looped Transformerscited by 12
  2. 2026Hyperloop Transformerscited by 11
  3. 2026Loop the Loopies!cited by 4
  4. 2026CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stabilitycited by 3
  5. 2026Fixed-Point Masked Generative Modelingcited by 2
  6. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Modelscited by 2
  7. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  8. 2026Recursive Scaling in Masked Diffusion Modelscited by 0
  9. 2026Looped Diffusion Transformercited by 0