mlsanityData to 7 Oct 2026

Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

arXiv 2403.09629 ↗First posted 14 Mar 2024cs.CL cs.AI cs.LGcited by 292

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 70 of its 92 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

70 of 92 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement12 Oct
  2. 2023Hypothesis Search: Inductive Reasoning with Language Models11 Sep
  3. 2023Learning to Reason and Memorize with Self-Notes1 May
  4. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions20 Dec
  5. 2022Explanations from Large Language Models Make Small Reasoners Better13 Oct
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 63

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  9. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  10. 2023Large Language Modelscited by 774
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  13. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  14. 2022Large Language Models Can Self-Improvecited by 656
  15. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  16. 2023Textbooks Are All You Needcited by 561
  17. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  18. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  19. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  20. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  21. 2022Large Language Models Are Reasoning Teacherscited by 318
  22. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  23. 2023Function Vectors in Large Language Modelscited by 292
  24. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  25. 2022Can language models learn from explanations in context?cited by 248
  26. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  27. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  28. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  29. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  30. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  31. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  32. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  33. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  34. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  35. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  36. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  37. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  38. 2022Language Models Can Teach Themselves to Program Bettercited by 75

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 64; 2025: 117; 2026: 111.

292 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2025s1: Simple test-time scalingcited by 1,311
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  5. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  6. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  7. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  8. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  9. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  10. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  11. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  12. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
38 more
  1. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  2. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  3. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2024Generative Reward Modelscited by 148
  6. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  10. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  11. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  12. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  13. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  14. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  15. 2025Recursive Language Modelscited by 94
  16. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  17. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 81
  18. 2026Privileged Information Distillation for Language Modelscited by 79
  19. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  20. 2024o1-Coder: an o1 Replication for Codingcited by 77
  21. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  22. 2025A Survey on Latent Reasoningcited by 71
  23. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  24. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  25. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  26. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  27. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  28. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  29. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  30. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  31. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  32. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  33. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  34. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  35. 2025Reinforcement Pre-Trainingcited by 40
  36. 2024Densing Law of LLMscited by 39
  37. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinementcited by 38
  38. 2025Step-GUI Technical Reportcited by 38

50 most cited of 292 All 292 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SCOTT: Self-Consistent Chain-of-Thought Distillation
  2. 2025Thinking Before You Speak: A Proactive Test-time Scaling Approach
  3. 2025Learning to Reason in LLMs by Expectation Maximization
  4. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation
  5. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking
  6. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
  7. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales
  8. 2025Reasoning Scaffolding: Distilling the Flow of Thought from LLMs