# Forking Paths in Neural Text Generation

arXiv: https://arxiv.org/abs/2412.07961 · HTML: https://mlsanity.com/paper/2412.07961/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2024.

## Built on by (most cited first, 30 of 34)

- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Rethinking On-Policy Self-Distillation for Thinking Models](https://mlsanity.com/paper/2607.05184/) (2026)
- [Reinforcing Video Reasoning with Focused Thinking](https://mlsanity.com/paper/2505.24718/) (2025)
- [Estimating LLM Uncertainty with Evidence](https://mlsanity.com/paper/2502.00290/) (2025)
- [On the Position Bias of On-Policy Distillation](https://mlsanity.com/paper/2606.22600/) (2026)
- [Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning](https://mlsanity.com/paper/2508.02260/) (2025)
- [Thought Branches: Interpreting LLM Reasoning Requires Resampling](https://mlsanity.com/paper/2510.27484/) (2025)
- [Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong](https://mlsanity.com/paper/2501.09775/) (2025)
- [Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics](https://mlsanity.com/paper/2511.04527/) (2025)
- [DISC: Dynamic Decomposition Improves LLM Inference Scaling](https://mlsanity.com/paper/2502.16706/) (2025)
- [Looking beyond the next token](https://mlsanity.com/paper/2504.11336/) (2025)
- [A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models](https://mlsanity.com/paper/2604.09921/) (2026)
- [Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selection](https://mlsanity.com/paper/2505.07309/) (2025)
- [APPO: Attention-guided Perception Policy Optimization for Video Reasoning](https://mlsanity.com/paper/2602.23823/) (2026)
- [When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy](https://mlsanity.com/paper/2605.12112/) (2026)
- [Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency](https://mlsanity.com/paper/2601.05905/) (2026)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2602.02103/) (2026)
- [Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models](https://mlsanity.com/paper/2508.06621/) (2025)
- [Disco-RAG: Discourse-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.04377/) (2026)
- [Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation](https://mlsanity.com/paper/2608.19611/) (2026)
- [It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories](https://mlsanity.com/paper/2609.03436/) (2026)
- [Revisiting the UID Hypothesis in LLM Reasoning Traces](https://mlsanity.com/paper/2510.13850/) (2025)
- [Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewards](https://mlsanity.com/paper/2510.23083/) (2025)
- [Embedding Perturbation may Better Reflect Intermediate-Step Uncertainty in LLM Reasoning](https://mlsanity.com/paper/2602.02427/) (2026)
- [Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms](https://mlsanity.com/paper/2606.08236/) (2026)
- [LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving](https://mlsanity.com/paper/2606.08470/) (2026)
- [ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization](https://mlsanity.com/paper/2607.05185/) (2026)
- [Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space](https://mlsanity.com/paper/2608.29188/) (2026)

## Abstract (arXiv)

Estimating uncertainty in Large Language Models (LLMs) is important for properly evaluating LLMs, and ensuring safety for users. However, prior approaches to uncertainty estimation focus on the final answer in generated text, ignoring intermediate steps that might dramatically impact the outcome. We hypothesize that there exist key forking tokens, such that re-sampling the system at those specific tokens, but not others, leads to very different outcomes. To test this empirically, we develop a novel approach to representing uncertainty dynamics across individual tokens of text generation, and applying statistical models to test our hypothesis. Our approach is highly flexible: it can be applied to any dataset and any LLM, without fine tuning or accessing model weights. We use our method to analyze LLM responses on 7 different tasks across 4 domains, spanning a wide range of typical use cases. We find many examples of forking tokens, including surprising ones such as punctuation marks, suggesting that LLMs are often just a single token away from saying something very different.