mlsanityData to 14 Oct 2026

OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

arXiv 2606.06096 ↗First posted 4 Jun 2026cs.LG cs.AI cs.CLcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-of-K discovery. We introduce OrderGrad, a family of likelihood-ratio and reparameterization gradient estimators for order-statistic objectives. OrderGrad optimizes finite-sample L-statistics, i.e., weighted averages of sorted rewards or costs, recovering objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights. For any fixed sample size and rank-weight vector, OrderGrad provides an unbiased gradient estimator for the corresponding order-statistic objective. The method is implemented as a simple reward transformation that can then be used in an otherwise standard policy-gradient or reparameterized update. We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective, including LLM math post-training and other tasks. OrderGrad provides a unified, plug-and-play route to risk-averse, robust, and exploratory learning. Code: https://github.com/paavo5/ordergrad

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 75 of its 112 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

75 of 112 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems21 May

Other references · 74

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
38 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  8. 2019Solving Rubik's Cube with a Robot Handcited by 734
  9. 2018Learning Dexterous In-Hand Manipulationcited by 687
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  12. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  13. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  14. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  15. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  16. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  17. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  18. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  19. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  20. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  21. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  22. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  23. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  24. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  25. 2016Variational inference for Monte Carlo objectivescited by 84
  26. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  27. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  28. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  29. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  30. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  31. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  32. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  33. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  34. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  35. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  36. 2017Learning with Average Top-k Losscited by 52
  37. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 51
  38. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tail-Likelihood Reinforcement Learningcited by 3
  2. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Post-Training with Policy Gradients: Optimality and the Base Model Barrier
  2. 2026Boosting CVaR Policy Optimization with Quantile Gradients
  3. 2026Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
  4. 2025Optimizing Return Distributions with Distributional Dynamic Programming
  5. 2024HAVER: Instance-Dependent Error Bounds for Maximum Mean Estimation and Applications to Q-Learning and Monte Carlo Tree Search
  6. 2026Efficient Best-of-N policy evaluation for inference-time alignment
  7. 2026Diverse Approaches to Optimal Execution Schedule Generation
  8. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradient