mlsanityData to 7 Oct 2026

LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

arXiv 2410.02884 ↗First posted 3 Oct 2024cs.AI cs.CLcited by 90

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

This paper presents an advanced mathematical problem-solving framework, LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS (SR-MCTS) overcomes the inefficiencies and limitations of conventional step-wise and greedy search algorithms by fostering a more efficient exploration of solution spaces. Pairwise Preference Reward Model~(PPRM), inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to model pairwise preferences between solutions, utilizing an Enhanced Borda Count (EBC) method to synthesize these preferences into a global ranking score to find better answers. This approach addresses the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability compared to existing methods like ToT and rStar, particularly in complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 67 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

53 of 67 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Step-level Value Preference Optimization for Mathematical Reasoning16 Jun
  2. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time25 May
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning1 May
  4. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  5. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning4 Mar
  6. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs26 Feb
  7. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset15 Feb
  8. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning9 Feb
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  10. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges31 Jan
  11. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline16 Jan
  12. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function1 Sep

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
29 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2022PAL: Program-aided Language Modelscited by 635
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  13. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  14. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  15. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  16. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  17. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  18. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  19. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  20. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  21. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  22. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  23. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  24. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  25. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  26. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  27. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  28. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  29. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 71; 2026: 8.

90 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  5. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  8. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  9. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  10. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  11. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  12. 2024o1-Coder: an o1 Replication for Codingcited by 77
38 more
  1. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  2. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  3. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  4. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  5. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  6. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  7. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  8. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 49
  9. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  10. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  11. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  12. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 39
  13. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinementcited by 38
  14. 2025O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoningcited by 34
  15. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  16. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  17. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  18. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  19. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  20. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  21. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  22. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  23. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  24. 2025BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoningcited by 19
  25. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  26. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  27. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  28. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 17
  29. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  30. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  31. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  32. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  33. 2025AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learningcited by 13
  34. 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processingcited by 12
  35. 2025Better Process Supervision with Bi-directional Rewarding Signalscited by 12
  36. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learningcited by 11
  37. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Searchcited by 11
  38. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B
  2. 2024MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Tree
  3. 2024Enhancing LLM Reasoning with Reward-guided Tree Search
  4. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
  5. 2025Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs
  6. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  7. 2024LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ
  8. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning