mlsanityData to 7 Oct 2026

Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models

arXiv 2602.05897 ↗First posted 5 Feb 2026cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings. However, they are prone to faithfulness hallucinations, especially in intermediate reasoning steps. Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained CoT evaluation, which can inadvertently reinforce unfaithful reasoning when the final answer is correct. To address these limitations, we propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model, together with an implicit truncated resampling strategy that generates contrastive signals from faithful prefixes, while also mitigating reward hacking from step-level rewards. Experiments across multiple SRMs and Open-Book QA benchmarks demonstrate that FaithRL consistently reduces hallucinations in both the CoT and final answers, leading to more faithful and reliable reasoning. Code is available at https://github.com/Easy195/FaithRL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 55 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

40 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  2. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning16 Oct
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2023Let's Verify Step by Step31 May

Other references · 36

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
24 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  4. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  5. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  6. 2025Why Language Models Hallucinatecited by 205
  7. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  8. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  9. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
  10. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  11. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  12. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 22
  13. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  14. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Modelscited by 18
  15. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  16. 2022Improving Simultaneous Machine Translation with Monolingual Datacited by 13
  17. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  18. 2025Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Modelscited by 9
  19. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 8
  20. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 7
  21. 2025SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generationcited by 5
  22. 2024Subgraph Aggregation for Out-of-Distribution Generalization on Graphscited by 4
  23. 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devicescited by 4
  24. 2025Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backendscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answeringcited by 4
  2. 2026CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Seriescited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization
  2. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  3. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
  4. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  5. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  6. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  7. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
  8. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning