mlsanityData to 7 Oct 2026

Pretraining with Token-Level Adaptive Latent Chain-of-Thought

arXiv 2602.08220 ↗First posted 9 Feb 2026cs.CLcited by 3

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-token computation without expanding parameters, by internalizing latent Chain-of-Thought (CoT) into pretraining. We propose Pretraining with Token-Level Adaptive Latent CoT (adaptive latent CoT), where the model generates a variable-length latent CoT trajectory before emitting each token -- allocating longer trajectories to difficult tokens and shorter (or even zero) trajectories to easy ones. Importantly, this behavior emerges naturally from one-stage pretraining on general text and reduces computation in both training and inference via token-wise adaptive halting. Experiments with Llama architectures show that adaptive latent CoT consistently improves language modeling perplexity and broad downstream accuracy, even with fewer training FLOPs than prior recurrent baselines.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 42 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

30 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning11 Nov
  2. 2025Scaling Latent Reasoning via Looped Language Models29 Oct
  3. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space27 Sep
  4. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking19 Feb
  5. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach7 Feb
  6. 2023Think before you speak: Training Language Models With Pause Tokens3 Oct

Other references · 24

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  12. 2024DeepSeek-V3 Technical Reportcited by 724
12 more
  1. 2018Universal Transformerscited by 613
  2. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  3. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  4. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  5. 2019Depth-Adaptive Transformercited by 227
  6. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  7. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  8. 2021PonderNet: Learning to Pondercited by 156
  9. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  10. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  11. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  12. 2023Computation vs. Communication Scaling for Future Transformers on Future Hardwarecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026HALO: Hybrid Adaptive Latent Reasoning for Language Modelscited by 0
  2. 2026SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrievalcited by 0
  3. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference
  2. 2026Latent Reasoning with Normalizing Flows
  3. 2025From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs
  4. 2024Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding
  5. 2023Training Chain-of-Thought via Latent-Variable Inference
  6. 2026LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification
  7. 2026Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
  8. 2026Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs