# Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning

arXiv: https://arxiv.org/abs/2605.04431 · HTML: https://mlsanity.com/paper/2605.04431/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 81 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 30 of 51)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](https://mlsanity.com/paper/2201.07284/) (2022)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [ODIN: Disentangled Reward Mitigates Hacking in RLHF](https://mlsanity.com/paper/2402.07319/) (2024)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023)
- [Language Models Resist Alignment: Evidence From Data Compression](https://mlsanity.com/paper/2406.06144/) (2024)
- [Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models](https://mlsanity.com/paper/2301.03797/) (2023)
- [Root Cause Analysis In Microservice Using Neural Granger Causal Discovery](https://mlsanity.com/paper/2402.01140/) (2024)
- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [AD-LLM: Benchmarking Large Language Models for Anomaly Detection](https://mlsanity.com/paper/2412.11142/) (2024)
- [CausalRCA: Causal Inference based Precise Fine-grained Root Cause Localization for Microservice Applications](https://mlsanity.com/paper/2209.02500/) (2022)
- [Xpert: Empowering Incident Management with Query Recommendations via Large Language Models](https://mlsanity.com/paper/2312.11988/) (2023)
- [Why Do Some Language Models Fake Alignment While Others Don't?](https://mlsanity.com/paper/2506.18032/) (2025)
- [X-lifecycle Learning for Cloud Incident Management using LLMs](https://mlsanity.com/paper/2404.03662/) (2024)
- [AgentFM: Role-Aware Failure Management for Distributed Databases with LLM-Driven Multi-Agents](https://mlsanity.com/paper/2504.06614/) (2025)
- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [ThinkFL: Self-Refining Failure Localization for Microservice Systems via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.18776/) (2025)
- [Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models](https://mlsanity.com/paper/2508.07173/) (2025)
- [d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models](https://mlsanity.com/paper/2512.09675/) (2025)
- [Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization](https://mlsanity.com/paper/2510.01555/) (2025)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)

## Built on by (most cited first, 3 of 3)

- [From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery](https://mlsanity.com/paper/2605.15412/) (2026)
- [Towards In-Depth Root Cause Localization for Microservices with Multi-Agent Recursion-of-Thought](https://mlsanity.com/paper/2605.14866/) (2026)
- [Playing social deduction games with reinforcement fine-tuned large language models](https://mlsanity.com/paper/2610.04261/) (2026)

## Abstract (arXiv)

Reinforcement fine-tuning (RFT) has become a core paradigm for post-training large language models, yet its training process remains highly fragile. Existing efforts mainly improve reliability at the system level or address specific issues in individual subproblems by modifying RFT algorithms. Despite their effectiveness, they largely overlook the problem of failure management at the training-process level. When training goes wrong, practitioners still rely heavily on expert-driven manual inspection and correction, and automatic failure management for RFT remains largely unexplored. In this paper, we take a first step toward systematic failure management for reinforcement fine-tuning. To understand the empirical structure of RFT failures, we first construct RFT-FaultBench, the first benchmark for fine-grained failures in reinforcement fine-tuning, covering 5 fault families, 16 fault types, 779 training runs, 22,549 train-step records, and 1,457,288 trajectory-level records. Based on this benchmark, we conduct a comprehensive empirical study showing that RFT failures are both observable from training dynamics and distinguishable through their empirical fault fingerprints. Building on these findings, we propose RFT-FM, an automatic failure management framework for reinforcement fine-tuning that unifies anomaly detection, failure diagnosis, and auto remediation in a closed loop. Experimental results show that RFT-FaultBench is neither trivial nor saturated: it exhibits clear anomaly structure while still posing substantial challenges, especially under subtle fault settings. Moreover, RFT-FM shows strong capability in detecting, diagnosing, and mitigating RFT failures.