mlsanityData to 14 Oct 2026

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

arXiv 2510.06499 ↗First posted 7 Oct 2025cs.CL cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success through imitation learning on vast text corpora, but this paradigm creates a training-generation gap and limits robust reasoning. Reinforcement learning (RL) offers a more data-efficient solution capable of bridging this gap, yet its application has been constrained by a critical data bottleneck: existing RL datasets are orders of magnitude smaller and less diverse than web-scale pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a scalable data engine that systematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RL. Using this pipeline, we construct the Webscale-RL dataset, containing 1.2 million examples across more than 9 domains. Our experiments show that the model trained on this dataset significantly outperforms continual pretraining and strong data refinement baselines across a suite of benchmarks. Notably, RL training with our dataset proves substantially more efficient, achieving the performance of continual pre-training with up to 100× fewer tokens. Our work presents a viable path toward scaling RL to pre-training levels, enabling more capable and efficient language models.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 61 of its 73 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

61 of 73 references.

References · 61

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  10. 2023Textbooks Are All You Needcited by 569
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  12. 2025Process Reinforcement through Implicit Rewardscited by 504
  13. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  14. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  15. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  16. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  17. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  18. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  19. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  20. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  21. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  22. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  23. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  24. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  25. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  26. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  27. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  28. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  29. 2024The pitfalls of next-token predictioncited by 160
  30. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  31. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  32. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  33. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  34. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  35. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  36. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  37. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  38. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  2. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  3. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  4. 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learningcited by 4
  5. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  6. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  7. 2026Learning Generalizable Behaviors for Terminal Agentscited by 2
  8. 2026CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  2. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  4. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  7. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
  8. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models