mlsanityData to 14 Oct 2026

UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection

arXiv 2505.12457 ↗First posted 18 May 2025cs.LG cs.CLcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Scaling RL for LLMs is computationally expensive, largely due to multi-sampling for policy optimization and evaluation, making efficient data selection crucial. Inspired by the Zone of Proximal Development (ZPD) theory, we hypothesize LLMs learn best from data within their potential comprehension zone. Addressing the limitation of conventional, computationally intensive multi-sampling methods for data assessment, we introduce UFO-RL. This novel framework uses a computationally efficient single-pass uncertainty estimation to identify informative data instances, achieving up to 185x faster data evaluation. UFO-RL leverages this metric to select data within the estimated ZPD for training. Experiments show that training with just 10% of data selected by UFO-RL yields performance comparable to or surpassing full-data training, reducing overall training time by up to 16x while enhancing stability and generalization. UFO-RL offers a practical and highly efficient strategy for scaling RL fine-tuning of LLMs by focusing learning on valuable data.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 28 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

22 of 28 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models3 Apr
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025LIMR: Less is More for RL Scaling17 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2023Let's Verify Step by Step31 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  11. 2025LIMO: Less is More for Reasoningcited by 498
  12. 2025A Survey on Post-training of Large Language Modelscited by 54
3 more
  1. 2024Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearningcited by 15
  2. 2025Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selectioncited by 7
  3. 2024Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patternscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinementcited by 3
  2. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimationcited by 2
  3. 2026Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoderscited by 2
  4. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  5. 2026A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPOcited by 0
  6. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient RLVR Training via Weighted Mutual Information Data Selection
  2. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  3. 2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment
  4. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization
  5. 2026Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR
  6. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
  7. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
  8. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian