mlsanityData to 7 Oct 2026

OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning

arXiv 2604.18530 ↗First posted 20 Apr 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have significantly improved Large Language Model (LLM) reasoning, yet models often struggle to explore novel trajectories beyond their initial policy distribution. While offline teacher guidance and entropy-driven strategies have been proposed to address this, they often lack deep integration or are constrained by the model's inherent capacity. In this paper, we propose OGER (Offline-Guided Exploration Reward), a novel framework that unifies offline teacher guidance and online reinforcement learning through a specialized reward modeling lens. OGER employs multi-teacher collaborative training and constructs an auxiliary exploration reward that leverages both offline trajectories and the model's own entropy to incentivize autonomous exploration. Extensive experiments across mathematical and general reasoning benchmarks demonstrate that OGER consistently outperforms competitive baselines, achieving substantial gains in mathematical reasoning while maintaining robust generalization to out-of-domain tasks. We provide a comprehensive analysis of training dynamics and conduct detailed ablation studies to validate the effectiveness of our entropy-aware reward modulation. Our code is available at https://github.com/ecoli-hit/OGER.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 40 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

26 of 40 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewarding16 Oct
  2. 2025Confidence as a Reward: Transforming LLMs into Reward Models15 Oct
  3. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning12 Oct
  4. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting9 Oct
  5. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training9 Oct
  6. 2025ExGRPO: Learning to Reason from Experience2 Oct
  7. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models2 Oct
  8. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  9. 2025REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning26 May
  10. 2025Learning to Reason under Off-Policy Guidance21 Apr
  11. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
3 more
  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  8. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  9. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  10. 2024DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimizationcited by 5
  11. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  2. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learning
  3. 2026Decoupling Exploration from Optimization in RLVR
  4. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  5. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration
  6. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
  7. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  8. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors