mlsanityData to 14 Oct 2026

Poly-EPO: Training Exploratory Reasoning Models

arXiv 2604.17654 ↗First posted 19 Apr 2026cs.AIcited by 5

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Exploration is a cornerstone of learning from experience: it enables agents to find solutions to complex problems, generalize to novel ones, and scale performance with test-time compute. In this paper, we present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation. The central idea is to train the LM to generate sets of responses that are collectively accurate under the reward function and exploratory in their reasoning strategies. We first develop a general recipe for optimizing LMs with set reinforcement learning (set RL) under arbitrary objective functions, showing how standard RL algorithms can be adapted to this setting through a modification to the advantage computation. We then propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation. Across a range of reasoning benchmarks, we show that Poly-EPO improves generalization, as evidenced by higher pass@k coverage, preserves greater diversity in model generations, and effectively scales with test-time compute.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 54 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

42 of 54 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs13 Jan
  2. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards18 Oct
  3. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models11 Sep
  4. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  5. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  6. 2025First Return, Entropy-Eliciting Explore9 Jul
  7. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  9. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  11. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
4 more
  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  12. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
14 more
  1. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  2. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  3. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  4. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  5. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  6. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  7. 2025Scaling Test-time Compute for LLM Agentscited by 63
  8. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  9. 2026Maximum Likelihood Reinforcement Learningcited by 36
  10. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  11. 2026Composer 2 Technical Reportcited by 22
  12. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  13. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  14. 2025Polychromic Objectives for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Dutch Books for Language Modelscited by 3
  2. 2026Tail-Likelihood Reinforcement Learningcited by 3
  3. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  4. 2026When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPOcited by 1
  5. 2026Exploration-Preserving Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
  2. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning
  3. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  4. 2025EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning
  5. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  6. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  7. 2026P^2O: Joint Policy and Prompt Optimization
  8. 2025COPO: Consistency-Aware Policy Optimization