mlsanityData to 7 Oct 2026

Tiny Autoregressive Recursive Models

arXiv 2603.08082 ↗First posted 9 Mar 2026cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state z and the predicted output y. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

36 of 43 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2014Neural Turing Machinescited by 630
  12. 2018Universal Transformerscited by 613
24 more
  1. 2019Deep Equilibrium Modelscited by 603
  2. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  3. 2014Memory Networkscited by 354
  4. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  5. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  6. 2019Depth-Adaptive Transformercited by 227
  7. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  8. 2019Levenshtein Transformercited by 197
  9. 2025Hierarchical Reasoning Modelcited by 167
  10. 2021PonderNet: Learning to Pondercited by 156
  11. 2020Multiscale Deep Equilibrium Modelscited by 149
  12. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  13. 2015Neural GPUs Learn Algorithmscited by 118
  14. 2015Neural Programmer-Interpreterscited by 96
  15. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  16. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  17. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  18. 2024Self-Healing Machine Learning: A Framework for Autonomous Adaptation in Real-World Environmentscited by 8
  19. 2025Statistical Hypothesis Testing for Auditing Robustness in Language Modelscited by 7
  20. 2026Tiny Recursive Reasoning with Mamba-2 Attention Hybridcited by 7
  21. 2025Deep Hierarchical Learning with Nested Subspace Networks for Large Language Modelscited by 5
  22. 2025Tiny Recursive Control: Iterative Reasoning for Efficient Optimal Controlcited by 4
  23. 2026ADEPT: Adaptive Dynamic Early-Exit Process for Transformerscited by 4
  24. 2026Tab-TRM: Tiny Recursive Model for Insurance Pricing on Tabular Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  2. 2026Loop the Loopies!cited by 4
  3. 2026Quantizing Recursive Reasoning Modelscited by 2
  4. 2026HALO: Hybrid Adaptive Latent Reasoning for Language Modelscited by 0
  5. 2026Recursive Vision Language Models for General Symbolic Reasoningcited by 0
  6. 2026Evaluating Tiny Recursive Models Across Training for Code Generationcited by 0