mlsanityData to 14 Oct 2026

Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning

arXiv 2609.35072 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We consider the problem of trajectory valuation in reinforcement learning: how to identify and mitigate detrimental trajectories during online training. Unlike classification, where data valuation relies on fixed training and validation sets, reinforcement learning involves dynamically generated trajectories without explicit validation signals, making conventional influence-based methods inapplicable. We propose Dynamic Trajectory Valuation (DTV), a simple and efficient framework that estimates trajectory utility at the mini-batch level and filters detrimental trajectories based solely on gradient information. By operating at the optimization level, DTV integrates seamlessly with existing reinforcement learning pipelines with minimal overhead. Extensive experiments across diverse settings, including PPO, GRPO, and DPO, demonstrate that DTV consistently improves performance, enhances data efficiency, and stabilizes optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 43 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

34 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning25 May

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2015Prioritized Experience Replaycited by 1,099
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
21 more
  1. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  2. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  3. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  4. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  6. 2020Prioritized Level Replaycited by 198
  7. 2025Generalizing Verifiable Instruction Followingcited by 194
  8. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  9. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  10. 2024Data Shapley in One Training Runcited by 81
  11. 2021Input Similarity from the Neural Network Perspectivecited by 79
  12. 2019Data Cleansing for Models Trained with SGDcited by 67
  13. 2023Data Pruning via Moving-one-Sample-outcited by 67
  14. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 12
  15. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  16. 2024Revisit, Extend, and Enhance Hessian-Free Influence Functionscited by 9
  17. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignmentcited by 9
  18. 2021Revisiting Prioritized Experience Replay: A Value Perspectivecited by 7
  19. 2026Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Trainingcited by 4
  20. 2025Z0-Inf: Zeroth Order Approximation for Data Influencecited by 3
  21. 2025Layer-Aware Influence for Online Data Valuation Estimationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation
  2. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
  3. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  4. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward
  5. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
  6. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents
  7. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  8. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learning