# A Novel Entropy-Maximizing TD3-based Reinforcement Learning for Automatic PID Tuning

arXiv: https://arxiv.org/abs/2210.02381 · HTML: https://mlsanity.com/paper/2210.02381/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2022.

## Built on by (most cited first, 2 of 2)

- [Real Time Self-Tuning Adaptive Controllers on Temperature Control Loops using Event-based Game Theory](https://mlsanity.com/paper/2506.13164/) (2025)
- [Methods for Multi-objective Optimization PID Controller for quadrotor UAVs](https://mlsanity.com/paper/2509.17423/) (2025)

## Abstract (arXiv)

Proportional-integral-derivative (PID) controllers have been widely used in the process industry. However, the satisfactory control performance of a PID controller depends strongly on the tuning parameters. Conventional PID tuning methods require extensive knowledge of the system model, which is not always known especially in the case of complex dynamical systems. In contrast, reinforcement learning-based PID tuning has gained popularity since it can treat PID tuning as a black-box problem and deliver the optimal PID parameters without requiring explicit process models. In this paper, we present a novel entropy-maximizing twin-delayed deep deterministic policy gradient (EMTD3) method for automating the PID tuning. In the proposed method, an entropy-maximizing stochastic actor is employed at the beginning to encourage the exploration of the action space. Then a deterministic actor is deployed to focus on local exploitation and discover the optimal solution. The incorporation of the entropy-maximizing term can significantly improve the sample efficiency and assist in fast convergence to the global solution. Our proposed method is applied to the PID tuning of a second-order system to verify its effectiveness in improving the sample efficiency and discovering the optimal PID parameters compared to traditional TD3.