# Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling

arXiv: https://arxiv.org/abs/2604.18464 · HTML: https://mlsanity.com/paper/2604.18464/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 13 of 13)

- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [ProcessBench: Identifying Process Errors in Mathematical Reasoning](https://mlsanity.com/paper/2412.06559/) (2024)
- [Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains](https://mlsanity.com/paper/2505.16552/) (2025)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures](https://mlsanity.com/paper/2509.14252/) (2025)
- [Next-Latent Prediction Transformers Learn Compact World Models](https://mlsanity.com/paper/2511.05963/) (2025)
- [LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals](https://mlsanity.com/paper/2604.05655/) (2026)
- [The Geometry of Reasoning: Flowing Logics in Representation Space](https://mlsanity.com/paper/2510.09782/) (2025)
- [Hidden States as Early Signals: Step-level Trace Evaluation and Pruning for Efficient Test-Time Scaling](https://mlsanity.com/paper/2601.09093/) (2026)
- [A Statistical Physics of Language Model Reasoning](https://mlsanity.com/paper/2506.04374/) (2025)
- [Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA](https://mlsanity.com/paper/2602.22617/) (2026)
- [Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability](https://mlsanity.com/paper/2603.10384/) (2026)
- [The Geometric Reasoner: Manifold-Informed Latent Foresight Search for Long-Context Reasoning](https://mlsanity.com/paper/2601.18832/) (2026)

## Built on by (most cited first, 1 of 1)

- [Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages](https://mlsanity.com/paper/2608.00533/) (2026)

## Abstract (arXiv)

Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168x more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4x for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3-12x over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2x more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.