# CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2602.00085 · HTML: https://mlsanity.com/paper/2602.00085/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning](https://mlsanity.com/paper/2507.00432/) (2025)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [Medical Hallucinations in Foundation Models and Their Impact on Healthcare](https://mlsanity.com/paper/2503.05777/) (2025)
- [Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.03318/) (2025)
- [Are Reasoning Models More Prone to Hallucination?](https://mlsanity.com/paper/2505.23646/) (2025)
- [Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing](https://mlsanity.com/paper/2402.00658/) (2024)

## Built on by (most cited first, 2 of 2)

- [Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces](https://mlsanity.com/paper/2604.11996/) (2026)
- [Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2610.02330/) (2026)

## Abstract (arXiv)

Reinforcement finetuning (RFT) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models. However, we identify a critical trade-off: while unconstrained RFT achieves strong reasoning performance, it severely compromises model trustworthiness by amplifying hallucination and worsening calibration; conversely, RKL-constrained RFT preserves trustworthiness but limits reasoning gains due to its unbounded penalty on exploratory deviations. To resolve this tension, we introduce CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), a novel method that replaces standard reverse KL regularization with a skew reverse KL divergence. CARE-RFT provides a confidence-sensitive penalty: it is bounded for confident, consistently rewarded explorations to enable reasoning, while unbounded elsewhere to preserve calibration. Extensive experiments across multiple model scales and RFT algorithms show that CARE-RFT achieves a superior balance, matching the reasoning performance of unconstrained RFT while recovering the trustworthiness and calibration of the base model. Our work establishes that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.