mlsanityData to 7 Oct 2026

AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition

arXiv 2402.11452 ↗First posted 18 Feb 2024cs.CLcited by 24

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework AutoPRM that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, AutoPRM first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to iteratively improve the subquestion solver. Additionally, we propose context-guided-decoding to avoid reward tampering and guide the subquestion solver towards the solution of the holistic problem. Extensive experiments show that AutoPRM significantly improves performance on mathematical and commonsense reasoning tasks over SOTA. More encouragingly, AutoPRM can be easily integrated with other orthogonal reasoning pipelines.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 35 of its 48 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

35 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning1 Aug
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  3. 2023Self-Evaluation Guided Beam Search for Reasoning1 May

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2023PaLM 2 Technical Reportcited by 1,154
20 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  7. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  13. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  14. 2023Fine-tuning Language Models for Factualitycited by 219
  15. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  16. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  17. 2023Word Embeddings Are Steers for Language Modelscited by 70
  18. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  19. 2024Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thoughtcited by 5
  20. 2017A Survey of Question Answering for Math and Science Problemcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 12; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  2. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  3. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  4. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  5. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  6. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  7. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  8. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  9. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  10. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  11. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  12. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
12 more
  1. 2024Editing Factual Knowledge and Explanatory Ability of Medical Large Language Modelscited by 10
  2. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  3. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  4. 2025From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualizationcited by 5
  5. 2024RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detectioncited by 4
  6. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4
  7. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modelingcited by 4
  8. 2025Mixture of Reasonings: Teach Large Language Models to Reason with Adaptive Strategiescited by 3
  9. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  10. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  11. 2026Provable and Practical In-Context Policy Optimization for Self-Improvementcited by 1
  12. 2025PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
  2. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
  3. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  4. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervision
  5. 2026DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution
  6. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
  7. 2025Self-Questioning Language Models
  8. 2025LADDER: Self-Improving LLMs Through Recursive Problem Decomposition