mlsanityData to 7 Oct 2026

Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models

arXiv 2607.05199 ↗First posted 6 Jul 2026cs.AIcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 44 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

37 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models18 Dec
  2. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents1 Dec

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
23 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  7. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  8. 2022Large Language Models Are Reasoning Teacherscited by 318
  9. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  10. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  11. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  12. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 76
  13. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  14. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  15. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  16. 2024Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulnesscited by 17
  17. 2023Using Large Language Model to Solve and Explain Physics Word Problems Approaching Human Levelcited by 15
  18. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  19. 2024Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedbackcited by 6
  20. 2025Simulus: Combining Improvements in Sample-Efficient World Model Agentscited by 6
  21. 2023CorrEmbed: Evaluating Pre-trained Model Image Similarity Efficacy with a Novel Metriccited by 1
  22. 2023Uncertain random geometric programming problemscited by 1
  23. 2025LMFCA-Net: A Lightweight Model for Multi-Channel Speech Enhancement with Efficient Narrow-Band and Cross-Band Attentioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decoupled Physical Modeling and Execution for Physics Reasoning
  2. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problems
  3. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Models
  4. 2025Dissecting Physics Reasoning in Small Language Models: A Multi-Dimensional Analysis from an Educational Perspective
  5. 2026PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems
  6. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
  7. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models
  8. 2026Reward Design for Physical Reasoning in Vision-Language Models