mlsanityData to 14 Oct 2026

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

arXiv 2511.19399 ↗First posted 24 Nov 2025cs.CL cs.AI cs.LGcited by 114

On the map: LLM agents and tool use → Web, search, and research agents

also in 1 more
Abstract (arXiv)

Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), where rubrics are constructed and maintained to co-evolve with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. First posted on arXiv 24 Nov 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 112.

114 papers.

Sort:

By area:

  1. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  2. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  3. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  5. 2026Aligning Language Models from User Interactionscited by 28
  6. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  7. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  8. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  9. 2026LLM Agents Already Know When to Call Tools -- Even Without Reasoningcited by 19
  10. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  11. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  12. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AIcited by 17
38 more
  1. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  2. 2026VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agentscited by 13
  3. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 13
  4. 2026Rubric-based On-policy Distillationcited by 13
  5. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 13
  6. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  7. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 11
  8. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 11
  9. 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factualitycited by 10
  10. 2026Scaling medical imaging report generation with multimodal reinforcement learningcited by 9
  11. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  12. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  13. 2026Training AI Scientists to Replicate Researchcited by 9
  14. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  15. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 8
  16. 2026SAGE: Benchmarking and Improving Retrieval for Deep Research Agentscited by 7
  17. 2026DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytellingcited by 7
  18. 2026Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generationcited by 7
  19. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  20. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraintscited by 7
  21. 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluationcited by 6
  22. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 6
  23. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  24. 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
  25. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 5
  26. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  27. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  28. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  29. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  30. 2026ProductResearch: Training E-Commerce Deep Research Agents via Multi-Agent Synthetic Trajectory Distillationcited by 3
  31. 2026Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agentcited by 3
  32. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 3
  33. 2026AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learningcited by 3
  34. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 3
  35. 2026SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Researchcited by 3
  36. 2026Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Rankingcited by 3
  37. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 3
  38. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 3

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents
  2. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification
  3. 2025Rethinking the Design of Reinforcement Learning-Based Deep Research Agents
  4. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
  5. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
  6. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search
  7. 2026Hybrid Open-Ended Tri-Evolution Makes Better Deep Researcher
  8. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards