# Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms

arXiv: https://arxiv.org/abs/2407.15283 · HTML: https://mlsanity.com/paper/2407.15283/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Reinforced Continual Learning](https://mlsanity.com/paper/1805.12369/) (2018)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings](https://mlsanity.com/paper/2008.06622/) (2020)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [Fast Adaptation via Policy-Dynamics Value Functions](https://mlsanity.com/paper/2007.02879/) (2020)
- [Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment](https://mlsanity.com/paper/2311.01059/) (2023)
- [Data-driven Machinery Fault Diagnosis: A Comprehensive Review](https://mlsanity.com/paper/2405.18843/) (2024)
- [Egocentric Visual Self-Modeling for Autonomous Robot Dynamics Prediction and Adaptation](https://mlsanity.com/paper/2207.03386/) (2022)
- [When Your Robot Breaks: Active Learning During Plant Failure](https://mlsanity.com/paper/1912.08116/) (2019)
- [Bio-Inspired Compensatory Strategies for Damage to Flapping Robotic Propulsors](https://mlsanity.com/paper/2402.01062/) (2024)

## Abstract (arXiv)

Industry is moving toward autonomous, network-connected machines that detect and adapt to changing conditions, including hardware faults. Conventional fault-tolerant design duplicates hardware and reroutes control logic; reinforcement learning (RL) offers a learning-based alternative. This paper presents the first systematic comparison of two RL algorithms -- Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) -- for integrating fault tolerance into control. Beyond algorithm choice, we investigate four knowledge-transfer strategies: retaining or discarding model parameters, and retaining or discarding storage contents. Performance is evaluated in two Gymnasium environments: Ant-v5 and FetchReachDense-v3. Results show rapid, fault-specific recovery with clear trade-offs. In Ant-v5, retaining PPO's parameters boosts early returns and remains the safest choice across all faults, while retaining SAC's parameters yields mixed outcomes. SAC's early performance further depends on whether the replay buffer is retained: beneficial when prior experiences match current dynamics, but harmful when they diverge. In FetchReachDense-v3, discarding both PPO's and SAC's parameters was most effective under sensor corruption. Across tasks, both algorithms recover near-normal performance within minutes in low-dimensional settings and within days in high-dimensional settings, highlighting a clear trade-off between adaptation speed and asymptotic performance. These findings demonstrate that RL can deliver robust fault tolerance and offer practical guidelines.