# Emergent Response Planning in LLMs

arXiv: https://arxiv.org/abs/2502.06258 · HTML: https://mlsanity.com/paper/2502.06258/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021)
- [LLM Internal States Reveal Hallucination Risk Faced With a Query](https://mlsanity.com/paper/2407.03282/) (2024)
- [Learning How Hard to Think: Input-Adaptive Allocation of LM Computation](https://mlsanity.com/paper/2410.04707/) (2024)
- [Do language models plan ahead for future tokens?](https://mlsanity.com/paper/2404.00859/) (2024)
- [Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models](https://mlsanity.com/paper/2405.19262/) (2024)
- [Do LLMs "know" internally when they follow instructions?](https://mlsanity.com/paper/2410.14516/) (2024)
- [HSF: Defending against Jailbreak Attacks with Hidden State Filtering](https://mlsanity.com/paper/2409.03788/) (2024)
- [Weakly Supervised Detection of Hallucinations in LLM Activations](https://mlsanity.com/paper/2312.02798/) (2023)

## Built on by (most cited first, 19 of 19)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://mlsanity.com/paper/2510.13554/) (2025)
- [SpecExit: Accelerating Large Reasoning Model via Speculative Exit](https://mlsanity.com/paper/2509.24248/) (2025)
- [Language Models Can Predict Their Own Behavior](https://mlsanity.com/paper/2502.13329/) (2025)
- [WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning](https://mlsanity.com/paper/2601.03164/) (2026)
- [Latent Planning Emerges with Scale](https://mlsanity.com/paper/2604.12493/) (2026)
- [From Simulation to Enaction: Post-trained language models recognize and react to their own generations](https://mlsanity.com/paper/2605.25459/) (2026)
- [The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steering](https://mlsanity.com/paper/2605.20745/) (2026)
- [How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs](https://mlsanity.com/paper/2606.10646/) (2026)
- [How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2602.02103/) (2026)
- [On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiency](https://mlsanity.com/paper/2601.21619/) (2026)
- [A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents](https://mlsanity.com/paper/2602.08964/) (2026)
- [How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding](https://mlsanity.com/paper/2603.18009/) (2026)
- [Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation](https://mlsanity.com/paper/2605.01846/) (2026)
- [Large Language Models Can Take False First Steps at Inference-time Planning](https://mlsanity.com/paper/2602.02991/) (2026)
- [From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond](https://mlsanity.com/paper/2606.28127/) (2026)
- [Signatures of semantic search in the activations of large language models](https://mlsanity.com/paper/2609.35599/) (2026)
- [To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model](https://mlsanity.com/paper/2610.02839/) (2026)
- [The belief-action gap in language model agents](https://mlsanity.com/paper/2610.17444/) (2026)

## Abstract (arXiv)

In this work, we argue that large language models (LLMs), though trained to predict only the next token, exhibit emergent planning behaviors: their hidden representations encode future outputs beyond the next token. Through simple probing, we demonstrate that LLM prompt representations encode global attributes of their entire responses, including textitstructure attributes (e.g., response length, reasoning steps), textitcontent attributes (e.g., character choices in storywriting, multiple-choice answers at the end of response), and textitbehavior attributes (e.g., answer confidence, factual consistency). In addition to identifying response planning, we explore how it scales with model size across tasks and how it evolves during generation. The findings that LLMs plan ahead for the future in their hidden representations suggest potential applications for improving transparency and generation control.