mlsanityData to 14 Oct 2026

Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction

arXiv 2509.12476 ↗First posted 15 Sep 2025cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Training a task-specific small reasoning model is challenging when direct human supervision or high-quality labels are scarce. However, LLMs with reasoning capabilities produce abundant intermediate reasoning traces that can be systematically refined to create effective supervision signals. We propose Reason-Refine-then-Align (R2tA), which turns refined model rationales into supervision for training task-specific reasoning models. Our method generates initial reasoning and responses from an open-source base model on task-specific inputs, then refines these traces, fixing hallucinations and inconsistencies, to form a high-fidelity dataset. We perform a two-stage alignment, supervised fine-tuning (SFT), followed by direct preference optimization (DPO) to calibrate the model's intermediate reasoning with human-validated conceptual preferences and then condition the final output on that aligned reasoning. As a case study, we apply R2tA to evaluate extended entity relationship diagrams (EERDs) in database system design, a structurally complex task where prompt-only methods miss or hallucinate errors. We curated a dataset of 600 EERD variants (train/test split of 450/150, respectively) with induced mistakes spanning 11 categories. Empirical evaluation suggests R2tA provides a practical, cost-effective path to scalable LLM adaptation in data-scarce domains, enabling reproducible AI tools for education and beyond.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 22 of its 47 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

22 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning14 Oct
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May

Other references · 19

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  4. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  5. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  6. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  7. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  8. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 40
  9. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  10. 2024Large Language Models for Education: A Surveycited by 34
  11. 2024Chain-of-Thought Unfaithfulness as Disguised Accuracycited by 28
  12. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
7 more
  1. 2024Keypoint-based Progressive Chain-of-Thought Distillation for LLMscited by 23
  2. 2023Weakly Supervised Detection of Hallucinations in LLM Activationscited by 20
  3. 2024LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loopcited by 8
  4. 2025Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Modelscited by 8
  5. 2025CoddLLM: Empowering Large Language Models for Data Analyticscited by 6
  6. 2024Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narrativescited by 5
  7. 2024PRISM: A Methodology for Auditing Biases in Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
  2. 2025ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
  3. 2023REFINER: Reasoning Feedback on Intermediate Representations
  4. 2025InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities
  5. 2026Compliance vs. Sensibility: On the Reasoning Controllability in Large Language Models
  6. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklists
  7. 2026Hint Tuning: Less Data Makes Better Reasoners
  8. 2023Making Large Language Models Better Reasoners with Alignment