mlsanityData to 7 Oct 2026

Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback

arXiv 2501.10799 ↗First posted 18 Jan 2025cs.LG cs.AIcited by 22

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have recently demonstrated remarkable success in mathematical reasoning. Despite progress in methods like chain-of-thought prompting and self-consistency sampling, these advances often focus on final correctness without ensuring that the underlying reasoning process is coherent and reliable. This paper introduces Step-KTO, a training framework that combines process-level and outcome-level binary feedback to guide LLMs toward more trustworthy reasoning trajectories. By providing binary evaluations for both the intermediate reasoning steps and the final answer, Step-KTO encourages the model to adhere to logical progressions rather than relying on superficial shortcuts. Our experiments on challenging mathematical benchmarks show that Step-KTO significantly improves both final answer accuracy and the quality of intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO achieves a notable improvement in Pass@1 accuracy over strong baselines. These results highlight the promise of integrating stepwise process feedback into LLM training, paving the way toward more interpretable and dependable reasoning capabilities.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 48 references that are in the map. First posted on arXiv 18 Jan 2025.

Builds on

38 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Building Math Agents with Multi-Turn Iterative Preference Learning4 Sep
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  3. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Math16 Feb
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  5. 2023Llemma: An Open Language Model For Mathematics16 Oct

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
21 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  10. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  11. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  12. 2023Fine-tuning Language Models for Factualitycited by 219
  13. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  14. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  15. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  16. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  17. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  18. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  19. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  20. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialoguecited by 32
  21. 2024Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generationscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 11.

22 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  3. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  4. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  5. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  6. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  7. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Contextcited by 10
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  9. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3
  10. 2026Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Rewardcited by 2
  11. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
  12. 2026CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Modelscited by 1
10 more
  1. 2026Bridging the Detection-to-Abstention Gap in Reasoning Models under Insufficient Informationcited by 1
  2. 2026When Model Merging Breaks Routing: Training-Free Calibration for MoEcited by 1
  3. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  4. 2025SIGMA: Search-Augmented On-Demand Knowledge Integration for Agentic Mathematical Reasoningcited by 0
  5. 2026GeoMind: An Agentic Workflow for Lithology Classification with Reasoned Tool Invocationcited by 0
  6. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoningcited by 0
  7. 2026Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Modelscited by 0
  8. 2026FlatManifold: Robust Continual Learning under Severe Label Noise and Domain Shifts via Intrinsic Manifold Flatteningcited by 0
  9. 2026Function-Level Execution Feedback for Code Preference Optimizationcited by 0
  10. 2026Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics
  2. 2024Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning
  3. 2024Enhancing Mathematical Reasoning in LLMs by Stepwise Correction
  4. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  5. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  6. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
  7. 2023Learning Multi-Step Reasoning by Solving Arithmetic Tasks
  8. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task