# FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning

arXiv: https://arxiv.org/abs/2410.06304 · HTML: https://mlsanity.com/paper/2410.06304/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning.
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Placed in

- LLM reasoning / [Hallucination detection and mitigation](https://mlsanity.com/lr/hallucination-detection-and-mitigation/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://mlsanity.com/paper/2304.10703/) (2023)
- [LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models](https://mlsanity.com/paper/2404.05221/) (2024)
- [FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge](https://mlsanity.com/paper/2305.08281/) (2023)
- [Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision](https://mlsanity.com/paper/2402.02658/) (2024)
- [SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation](https://mlsanity.com/paper/2310.00074/) (2023)
- [ThoughtSource: A central hub for large language model reasoning data](https://mlsanity.com/paper/2301.11596/) (2023)
- [Retrieving Supporting Evidence for LLMs Generated Answers](https://mlsanity.com/paper/2306.13781/) (2023)
- [Measuring and Modifying Factual Knowledge in Large Language Models](https://mlsanity.com/paper/2306.06264/) (2023)

## Built on by (most cited first, 12 of 12)

- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users](https://mlsanity.com/paper/2508.17281/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs](https://mlsanity.com/paper/2503.02003/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2502.01812/) (2025)
- [Adversarial Training for Process Reward Models](https://mlsanity.com/paper/2511.22888/) (2025)
- [Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling](https://mlsanity.com/paper/2506.00064/) (2025)
- [Evaluating Large Language Models on Solved and Unsolved Problems in Graph Theory: Implications for Computing Education](https://mlsanity.com/paper/2602.05059/) (2026)
- [SourceBench: Can AI Answers Reference Quality Web Sources?](https://mlsanity.com/paper/2602.16942/) (2026)
- [R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning](https://mlsanity.com/paper/2603.25720/) (2026)
- [Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM](https://mlsanity.com/paper/2607.18915/) (2026)

## Abstract (arXiv)

Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as mathematical problem-solving. Existing approaches primarily detect the presence of hallucinations but lack a nuanced understanding of their types and manifestations. In this paper, we first introduce a comprehensive taxonomy that categorizes the common hallucinations in mathematical reasoning tasks into six types. We then propose FG-PRM (Fine-Grained Process Reward Model), an augmented model designed to detect and mitigate hallucinations in a fine-grained, step-level manner. To address the limitations of manually labeling training data, we propose an automated method for generating fine-grained hallucination data using LLMs. Our FG-PRM demonstrates superior performance across two key tasks: 1) Fine-grained hallucination detection: classifying hallucination types for each reasoning step; and 2) Verification: ranking multiple LLM-generated outputs to select the most accurate solution. Our experiments show that FG-PRM excels in fine-grained hallucination detection and substantially boosts the performance of LLMs on GSM8K and MATH benchmarks. These results highlight the benefits of fine-grained supervision in enhancing the reliability and interpretability of LLM reasoning processes.