mlsanityData to 7 Oct 2026

BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

arXiv 2608.02867 ↗First posted 3 Aug 2026cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 40 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

26 of 40 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model21 Jun
  2. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtraining13 Jun
  3. 2026SD-E2: Semantic Exploration for Reasoning Under Token Budgets25 Jan
  4. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs13 Jan
  5. 2026Reinforced Efficient Reasoning via Semantically Diverse Exploration8 Jan
  6. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  7. 2025Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains20 Oct
  8. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  9. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  10. 2025Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning28 Sep
  11. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  12. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
7 more
  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning14 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 7

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2023A Configurable Library for Generating and Manipulating Maze Datasetscited by 31
  6. 2025What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoningcited by 21
  7. 2025KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  2. 2026Reasoning Concentrates Errors, and Self-Consistency Never Noticescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  4. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  5. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  6. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs