mlsanityData to 14 Oct 2026

Tulu 3: Pushing Frontiers in Open Language Model Post-Training

arXiv 2411.15124 ↗First posted 22 Nov 2024cs.CLcited by 1,053

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains.

Key facts. Cited by 1,053 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 104 of its 131 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

104 of 131 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 99

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2024GPT-4o System Cardcited by 4,588
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2023Mistral 7Bcited by 3,164
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  14. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  15. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  16. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  17. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  18. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  19. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  20. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  21. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  22. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  23. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  24. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  25. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  26. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  27. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  28. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  29. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  30. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  31. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  32. 2024Yi: Open Foundation Models by 01.AIcited by 719
  33. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  34. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  35. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  36. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  37. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  38. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 458; 2026: 583.

1,053 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  4. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  5. 2025Process Reinforcement through Implicit Rewardscited by 504
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  7. 2025Muon is Scalable for LLM Trainingcited by 428
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  10. 2026Reinforcement Learning via Self-Distillationcited by 332
  11. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  12. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
38 more
  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  4. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  6. 2025Kimi-Audio Technical Reportcited by 222
  7. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  8. 2025Learning to Reason without External Rewardscited by 210
  9. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  10. 2025Generalizing Verifiable Instruction Followingcited by 194
  11. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  12. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  13. 2025Agentic Reinforced Policy Optimizationcited by 172
  14. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  15. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  16. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  17. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  18. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  19. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  20. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  21. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  22. 2025Reinforcement Learning with Rubric Anchorscited by 113
  23. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  24. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 110
  25. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  26. 2025Agentic Large Language Models, a surveycited by 108
  27. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  28. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  29. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  30. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  31. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  32. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  33. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  34. 2025Reinforcing General Reasoning without Verifierscited by 77
  35. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 76
  36. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  37. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  38. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71

50 most cited of 1,053 All 1,053 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  2. 2025Towards a Unified View of Large Language Model Post-Training
  3. 2025EvoLM: In Search of Lost Language Model Training Dynamics
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  5. 2024Reinforcement Learning for LLM Post-Training: A Survey
  6. 2025Post-Completion Learning for Language Models
  7. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  8. 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution