mlsanityData to 14 Oct 2026

CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference

arXiv 2310.10845 ↗First posted 16 Oct 2023cs.CL cs.LGcited by 27

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Scaling language models to larger and deeper sizes has led to significant boosts in performance. Even though the size of these models limits their application in compute-constrained environments, the race to continually develop ever larger and deeper foundational models is underway. At the same time -- regardless of the model size -- task-specific techniques continue to play a pivotal role in achieving optimal downstream performance. One of these techniques, called Chain-of-Thought (CoT), is particularly interesting since, as we point out in this work, it resembles employing a deeper transformer through re-applying the model multiple times. However, a key subtlety in computing the attention of past tokens differentiates CoT from simply applying the model several times. Based on this insight, we propose CoTFormer, a novel architecture which closely mimics CoT at the token level, allowing us to obtain significantly improved accuracies close to much larger models. While applying CoT introduces additional computation costs, we compensate for it by leveraging CoTFormer's special compatibility with token-wise variable depth. Through a compute adaptive model -- which automatically allocates the compute to tokens that need it most -- we show that it is possible to reduce the computation cost significantly without any reduction in accuracy, and with further compute cost reductions possible while maintaining a competitive accuracy.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 28 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

23 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step24 Jun
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2020On Layer Normalization in the Transformer Architecturecited by 922
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  8. 2020Efficient Transformers: A Surveycited by 790
  9. 2023Large Language Modelscited by 780
  10. 2018Universal Transformerscited by 647
  11. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  12. 2018Slimmable Neural Networkscited by 305
8 more
  1. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  2. 2019Depth-Adaptive Transformercited by 236
  3. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  4. 2017Adaptive Neural Networks for Efficient Inferencecited by 163
  5. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  6. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  7. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 25
  8. 2021Masked Training of Neural Networks with Partial Gradientscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 15.

27 papers.

Sort:

By area:

  1. 2025A Survey on Latent Reasoningcited by 81
  2. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  3. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  4. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  5. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Modelscited by 21
  6. 2024Rational Metareasoning for Large Language Modelscited by 20
  7. 2026Generative Recursive Reasoningcited by 20
  8. 2026Training-Free Looped Transformerscited by 15
  9. 2025Parallel Test-Time Scaling for Latent Reasoning Modelscited by 14
  10. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12
  11. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  12. 2026LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modelingcited by 10
15 more
  1. 2026Are Latent Reasoning Models Easily Interpretable?cited by 9
  2. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  3. 2026Loop the Loopies!cited by 8
  4. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  5. 2026SeLaR: Selective Latent Reasoning in Large Language Modelscited by 7
  6. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 6
  7. 2025Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lenscited by 2
  8. 2025Towards Inference-time Scaling for Continuous Space Reasoningcited by 2
  9. 2026M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detectioncited by 1
  10. 2026Looped Diffusion Language Modelscited by 1
  11. 2026Reproducing Recurrent Transformers: The CoTFormercited by 1
  12. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1
  13. 2026Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memorycited by 0
  14. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0
  15. 2026Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding
  2. 2026Pretraining with Token-Level Adaptive Latent Chain-of-Thought
  3. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
  4. 2025Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
  5. 2025CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
  6. 2025From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement
  7. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  8. 2025Reasoning Efficiently Through Adaptive Chain-of-Thought Compression: A Self-Optimizing Framework