mlsanityData to 14 Oct 2026

HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs

arXiv 2509.23967 ↗First posted 28 Sep 2025cs.CLcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) increasingly rely on Chain-of-Thought (CoT) reasoning to improve accuracy on complex tasks. However, always generating lengthy reasoning traces is inefficient, leading to excessive token usage and higher inference costs. This paper introduces the Hybrid Policy Optimization (i.e., HiPO), a framework for adaptive reasoning control that enables LLMs to selectively decide when to engage in detailed reasoning (Think-on) and when to respond directly (Think-off). Specifically, HiPO combines a hybrid data pipelineproviding paired Think-on and Think-off responseswith a hybrid reinforcement learning reward system that balances accuracy and efficiency while avoiding over-reliance on detailed reasoning. Experiments across mathematics and coding benchmarks demonstrate that HiPO can substantially reduce token length while maintaining or improving accuracy. Finally, we hope HiPO a can be a principled approach for efficient adaptive reasoning, advancing the deployment of reasoning-oriented LLMs in real-world, resource-sensitive settings.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 44 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

39 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025Training Language Models to Reason Efficiently6 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2023Let's Verify Step by Step31 May
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2024DeepSeek-V3 Technical Reportcited by 811
  9. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  10. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  11. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  12. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
19 more
  1. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  2. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 249
  3. 2025Dynamic Early Exit in Reasoning Modelscited by 219
  4. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  5. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  6. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 169
  7. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 158
  8. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 147
  9. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 117
  10. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  11. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 109
  12. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 99
  13. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 96
  14. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 87
  15. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  16. 2024Can Language Models Learn to Skip Steps?cited by 59
  17. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  18. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matterscited by 25
  19. 2025KAT-V1: Kwai-AutoThink Technical Reportcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 14
  2. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Modelscited by 11
  3. 2026PseudoAct: Leveraging Pseudocode Synthesis for Flexible Planning and Action Control in Large Language Model Agentscited by 2
  4. 2026AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answeringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
  2. 2025LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
  3. 2025Think Only When You Need with Large Hybrid-Reasoning Models
  4. 2025Thinkless: LLM Learns When to Think
  5. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning
  6. 2026Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
  7. 2026Hint-Guided Diversified Policy Optimization for LLM Reasoning
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression