mlsanityData to 7 Oct 2026

Advancing LLM Reasoning Generalists with Preference Trees

arXiv 2404.02078 ↗First posted 2 Apr 2024cs.AI cs.CL cs.LGcited by 191

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks covering mathematics, code generation, and logical reasoning problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a comprehensive benchmarking across 12 tests covering five tasks, and achieves a 33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging benchmarks, substantially outperforming existing open-source models by margins more than 13.3%. The strong performance of Eurus can be primarily attributed to UltraInteract, our newly-curated large-scale, high-quality alignment dataset specifically designed for complex reasoning tasks. UltraInteract can be used in both supervised fine-tuning and preference learning. For each instruction, it includes a preference tree consisting of (1) reasoning chains with diverse planning strategies in a unified format, (2) multi-turn interaction trajectories with the environment and the critique, and (3) pairwise data to facilitate preference learning. UltraInteract allows us to conduct an in-depth exploration of preference learning for reasoning tasks. Our investigation reveals that some well-established preference learning algorithms may be less suitable for reasoning tasks compared to their effectiveness in general conversations. Inspired by this, we derive a novel reward modeling objective which, together with UltraInteract, leads to a strong reward model.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 50 of its 58 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

50 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism5 Jan
  2. 2023Code Llama: Open Foundation Models for Code24 Aug
  3. 2023Specializing Smaller Language Models towards Multi-Step Reasoning30 Jan

Other references · 47

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
35 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  13. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  14. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  15. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  16. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  17. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  18. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  19. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  20. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  21. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  22. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  23. 2023Generative Judge for Evaluating Alignmentcited by 163
  24. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  25. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  26. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  27. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  28. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  29. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  30. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  31. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  32. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 97
  33. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  34. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  35. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 87; 2026: 33.

191 papers.

Sort:

By area:

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2025Process Reinforcement through Implicit Rewardscited by 478
  4. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  5. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  7. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  9. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  11. 2024Generative Reward Modelscited by 148
  12. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
38 more
  1. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  2. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  3. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2025Geometric-Mean Policy Optimizationcited by 107
  6. 2024Critique-out-Loud Reward Modelscited by 105
  7. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  8. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  10. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  11. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  12. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  13. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  14. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  15. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  16. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  17. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  18. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  19. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  20. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  21. 2025Self-rewarding correction for mathematical reasoningcited by 67
  22. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  23. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  24. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  25. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  26. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  27. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  28. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  29. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  30. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  31. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  32. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  33. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  34. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  35. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  36. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  37. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  38. 2024Entropy-Regularized Process Reward Modelcited by 38

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VERUS-LM: a Versatile Framework for Combining LLMs with Symbolic Reasoning
  2. 2024Advancing Process Verification for Large Language Models via Tree-Based Preference Learning
  3. 2026How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
  4. 2024LLMs for Relational Reasoning: How Far are We?
  5. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  7. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
  8. 2024Eliciting Better Multilingual Structured Reasoning from LLMs through Code