# ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

arXiv: https://arxiv.org/abs/2608.03972 · HTML: https://mlsanity.com/paper/2608.03972/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 38 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 30 of 36)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2310.03731/) (2023)
- [Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning](https://mlsanity.com/paper/2505.17813/) (2025)
- [PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection](https://mlsanity.com/paper/2502.12119/) (2025)
- [UFT: Unifying Supervised and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.16984/) (2025)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process](https://mlsanity.com/paper/2505.13408/) (2025)
- [ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM](https://mlsanity.com/paper/2506.14766/) (2025)
- [Backdoor Cleaning without External Guidance in MLLM Fine-tuning](https://mlsanity.com/paper/2505.16916/) (2025)
- [Self-Evolving Multi-Agent Systems via Textual Backpropagation](https://mlsanity.com/paper/2506.09046/) (2025)
- [SPOT! Revisiting Video-Language Models for Event Understanding](https://mlsanity.com/paper/2311.12919/) (2023)
- [Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers](https://mlsanity.com/paper/2509.03059/) (2025)
- [Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models](https://mlsanity.com/paper/2511.02650/) (2025)
- [Beyond Magic Words: Sharpness-Aware Prompt Evolving for Robust Large Language Models with TARE](https://mlsanity.com/paper/2509.24130/) (2025)

## Built on by (most cited first, 6 of 6)

- [OPD-V: Visual On-Policy Self-Distillation with Modality Balance](https://mlsanity.com/paper/2608.05131/) (2026)
- [My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.01161/) (2026)
- [VERA: Scaling Verifiable Environments for Agentic co-Evolution](https://mlsanity.com/paper/2610.05923/) (2026)
- [VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction](https://mlsanity.com/paper/2610.06293/) (2026)
- [X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning](https://mlsanity.com/paper/2610.16811/) (2026)
- [Agent-Reflex: Teaching Language Agents to Act Reflectively](https://mlsanity.com/paper/2610.17953/) (2026)

## Abstract (arXiv)

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.