Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168x more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4x for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3-12x over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2x more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 13 of its 22 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
13 of 22 references.
Closest ancestors · 6
Same direction, newest first
- 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals7 Apr
- 2026Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability11 Mar
- 2026The Geometric Reasoner: Manifold-Informed Latent Foresight Search for Long-Context Reasoning25 Jan
- 2026Hidden States as Early Signals: Step-level Trace Evaluation and Pruning for Efficient Test-Time Scaling14 Jan
- 2025A Statistical Physics of Language Model Reasoning4 Jun
- 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains22 May
Other references · 7
- 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
- 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
- 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
- 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
- 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
- 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
- 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Geometric Latent Reasoning Induces Shorter Generations in LLMs
- 2025TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Model
- 2026How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning
- 2026Reasoning-Aware Training for Time Series Forecasting
- 2026GeoSteer: Faithful Chain-of-Thought Steering via Latent Manifold Gradients
- 2026How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories
- 2026A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
- 2026Invariant Reasoning Directions in Latent Trajectories of Language Models