mlsanityData to 14 Oct 2026

Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

arXiv 2311.10702 ↗First posted 17 Nov 2023cs.CLcited by 229

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Since the release of T"ULU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into T"ULU, resulting in T"ULU 2, a suite of improved T"ULU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) T"ULU-V2-mix, an improved collection of high-quality instruction datasets; (2) T"ULU 2, LLAMA-2 models finetuned on the V2 mixture; (3) T"ULU 2+DPO, T"ULU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (T"ULU 2+DPO 70B); (4) CODE T"ULU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the T"ULU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data, training and evaluation code to facilitate future open efforts on adapting large language models.

Key facts. Cited by 229 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 46 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

32 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
17 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  6. 2023Instruction Tuning with GPT-4cited by 686
  7. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  9. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  11. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  13. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  14. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  15. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  16. 2019Inferring Which Medical Treatments Work from Reports of Clinical Trialscited by 64
  17. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 142; 2025: 64; 2026: 19.

229 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  6. 2024LoRA Learns Less and Forgets Lesscited by 328
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  9. 2024Generative Representational Instruction Tuningcited by 233
  10. 2025Generalizing Verifiable Instruction Followingcited by 194
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  12. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
38 more
  1. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  2. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  4. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  5. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  6. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  7. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  8. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  9. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  10. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  11. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  12. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  13. 2024Tuning Language Models by Proxycited by 89
  14. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  15. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  16. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  17. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
  18. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  19. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  20. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  21. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  22. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  23. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  24. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  25. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  26. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  27. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 62
  28. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  29. 2025Reinforcement Learning from Human Feedbackcited by 58
  30. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  31. 2024Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversationcited by 55
  32. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  33. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  34. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  35. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 51
  36. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  37. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  38. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48

50 most cited of 229 All 229 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
  2. 2023Tuna: Instruction Tuning using Feedback from Large Language Models
  3. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
  4. 2026GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
  5. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  6. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  7. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLM
  8. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning