mlsanityData to 7 Oct 2026

Think before you speak: Training Language Models With Pause Tokens

arXiv 2310.02226 ↗First posted 3 Oct 2023cs.CL cs.AI cs.LGcited by 254

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Language models generate responses by producing a series of tokens in immediate succession: the (K+1)th token is an outcome of manipulating K hidden vectors per layer, one vector per preceding token. What if instead we were to let the model manipulate say, K+10 hidden vectors, before it outputs the (K+1)th token? We operationalize this idea by performing training and inference on language models with a (learnable) textitpause token, a sequence of which is appended to the input prefix. We then delay extracting the model's outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluate textitpause-training on decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall. Our main finding is that inference-time delays show gains when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on 8 of 9 tasks, most prominently, a gain of 18% EM score on the QA task of SQuAD, 8% on CommonSenseQA and 1% accuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 49 of its 67 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

49 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  3. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  4. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
33 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2023Vision Transformers Need Registerscited by 844
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2018CoQA: A Conversational Question Answering Challengecited by 674
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  11. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  12. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  13. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  14. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  15. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  16. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  17. 2022Recurrent Memory Transformercited by 197
  18. 2021PonderNet: Learning to Pondercited by 156
  19. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  20. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  21. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  22. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  23. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  24. 2019Augmenting Self-attention with Persistent Memorycited by 92
  25. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  26. 2020Memory Transformercited by 74
  27. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  28. 2023Learning to Reason and Memorize with Self-Notescited by 30
  29. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  30. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  31. 2020An Embarrassingly Simple Model for Dialogue Relation Extractioncited by 13
  32. 2021DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inferencecited by 9
  33. 2020Few-shot Sequence Learning with Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 51; 2025: 99; 2026: 102.

254 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  6. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  7. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  8. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  9. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  10. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  11. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  12. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
38 more
  1. 2024The pitfalls of next-token predictioncited by 156
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  5. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 115
  6. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  7. 2025SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMscited by 110
  8. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  9. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  10. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  11. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 81
  12. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  13. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 73
  14. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  15. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  16. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  17. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
  18. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  19. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  20. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  21. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  22. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  23. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 56
  24. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  25. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  26. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  27. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  28. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  29. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 46
  30. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  31. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
  32. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  33. 2024Denoising Vision Transformerscited by 40
  34. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 40
  35. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  36. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  37. 2025LLM Pretraining with Continuous Conceptscited by 33
  38. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32

50 most cited of 254 All 254 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning to Skip for Language Modeling
  2. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference
  3. 2024Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens
  4. 2026Language-Conditioned Token and Reasoning Efficiency in Large Language Models: A Paired Cross-Lingual Study Protocol
  5. 2026Distribution-Aware Companding Quantization of Large Language Models
  6. 2026Enhancing reasoning accuracy in large language models during inference time
  7. 2021Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning
  8. 2025Fast Quiet-STaR: Thinking Without Thought Tokens