mlsanityData to 14 Oct 2026

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

arXiv 2606.24994 ↗First posted 23 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 13 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

9 of 13 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 5

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  2. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  3. 2025Reasoning with Exploration: An Entropy Perspective
  4. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
  5. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  7. 2026Gradient Extrapolation-Based Policy Optimization
  8. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities