mlsanityData to 14 Oct 2026

Tail-Likelihood Reinforcement Learning

arXiv 2609.02987 ↗First posted 2 Sep 2026cs.LG stat.MLcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 105 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

40 of 105 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation4 Jun
  2. 2026Delightful Policy Gradient15 Mar
  3. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapse23 Oct

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2024OpenAI o1 System Cardcited by 1,244
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
25 more
  1. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  2. 2022Defining and Characterizing Reward Hackingcited by 203
  3. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  4. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114
  5. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  6. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  7. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  8. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  9. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  10. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 78
  11. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 77
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  13. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 43
  14. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  15. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  16. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  17. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
  18. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 18
  19. 2025What is the objective of reasoning with reinforcement learning?cited by 18
  20. 2020The gem5 Simulator: Version 20.0+cited by 14
  21. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 10
  22. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  23. 2026Target Policy Optimizationcited by 1
  24. 2026ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimizationcited by 1
  25. 2026RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihoodcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Matryoshka attribution: Learning to attribute language model outputs to representations and weightscited by 1
  2. 2026EasyPPO: Stabilizing the Critic Is Keycited by 0
  3. 2026Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Maximum Likelihood Reinforcement Learning
  2. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  3. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
  4. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performance
  5. 2026On Training in Imagination
  6. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  7. 2026Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
  8. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning