# Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents

arXiv: https://arxiv.org/abs/2412.00821 · HTML: https://mlsanity.com/paper/2412.00821/

## Key facts

- Cited by 23 papers in the map (data through 13 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning](https://mlsanity.com/paper/2308.00436/) (2023)
- [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2310.03731/) (2023)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models](https://mlsanity.com/paper/2305.15074/) (2023)
- [SciAgent: Tool-augmented Language Models for Scientific Reasoning](https://mlsanity.com/paper/2402.11451/) (2024)
- [Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction](https://mlsanity.com/paper/2406.00755/) (2024)
- [Structured Chemistry Reasoning with Large Language Models](https://mlsanity.com/paper/2311.09656/) (2023)
- [Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models](https://mlsanity.com/paper/2304.11657/) (2023)
- [Rethinking Language Models as Symbolic Knowledge Graphs](https://mlsanity.com/paper/2308.13676/) (2023)
- [Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks](https://mlsanity.com/paper/2404.13099/) (2024)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)

## Built on by (most cited first, 23 of 23)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models](https://mlsanity.com/paper/2502.00334/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics](https://mlsanity.com/paper/2502.15815/) (2025)
- [PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaboration](https://mlsanity.com/paper/2505.15047/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning](https://mlsanity.com/paper/2510.03185/) (2025)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [S1-NexusAgent: a Self-Evolving Agent Framework for Multidisciplinary Scientific Research](https://mlsanity.com/paper/2602.01550/) (2026)
- [A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities](https://mlsanity.com/paper/2507.08425/) (2025)
- [ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis](https://mlsanity.com/paper/2604.16922/) (2026)
- [Mastering Olympiad-Level Physics with Artificial Intelligence](https://mlsanity.com/paper/2511.10515/) (2025)
- [STEMVerse: A Dual-Axis Diagnostic Framework for STEM Reasoning in Large Language Models](https://mlsanity.com/paper/2602.02497/) (2026)
- [Decoupled Physical Modeling and Execution for Physics Reasoning](https://mlsanity.com/paper/2608.22126/) (2026)
- [SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code](https://mlsanity.com/paper/2512.05954/) (2025)
- [You Snooze, You Lose: Automatic Safety Alignment Restoration through Neural Weight Translation](https://mlsanity.com/paper/2605.04992/) (2026)
- [When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoning](https://mlsanity.com/paper/2605.06772/) (2026)
- [Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds](https://mlsanity.com/paper/2605.18827/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models](https://mlsanity.com/paper/2607.05199/) (2026)
- [FlatManifold: Robust Continual Learning under Severe Label Noise and Domain Shifts via Intrinsic Manifold Flattening](https://mlsanity.com/paper/2607.05201/) (2026)
- [OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora](https://mlsanity.com/paper/2608.25398/) (2026)
- [Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning](https://mlsanity.com/paper/2609.18991/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) demonstrate remarkable capabilities in various reasoning tasks. However, they encounter significant challenges when it comes to scientific reasoning, particularly in physics, which requires not only mathematical reasoning but also factual and conceptual understanding. When addressing complex physics problems, LLMs typically face three key issues: problem miscomprehension, incorrect concept application, and computational errors. While each of these problems can be addressed individually, there is a need for a generalized approach that can tackle all three issues simultaneously. To address this, we introduce Mixture of Refinement Agents (MoRA), a novel agentic refinement framework that iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. Our approach aims to bridge the gap between opensource LLMs and GPT-4o by utilizing the latter as error identifier to guide these refinement agents. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA). MoRA significantly improves the performance of Llama-3-70B and Gemma-2-27B on these datasets, achieving up to a 16% increase in final answer accuracy.