# Direct and indirect reinforcement learning

arXiv: https://arxiv.org/abs/1912.10600 · HTML: https://mlsanity.com/paper/1912.10600/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 11 of 11)

- [Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model](https://mlsanity.com/paper/2102.11513/) (2021)
- [Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process](https://mlsanity.com/paper/2405.11870/) (2024)
- [Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions](https://mlsanity.com/paper/2203.02605/) (2022)
- [Training Multi-layer Neural Networks on Ising Machine](https://mlsanity.com/paper/2311.03408/) (2023)
- [Decision-Making under On-Ramp merge Scenarios by Distributional Soft Actor-Critic Algorithm](https://mlsanity.com/paper/2103.04535/) (2021)
- [Approximate Optimal Filter for Linear Gaussian Time-invariant Systems](https://mlsanity.com/paper/2103.05505/) (2021)
- [Artificial Intelligence-based Decision Support Systems for Precision and Digital Health](https://mlsanity.com/paper/2407.16062/) (2024)
- [Predictive Lagrangian Optimization for Constrained Reinforcement Learning](https://mlsanity.com/paper/2501.15217/) (2025)
- [FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2606.21587/) (2026)
- [NeuralParker: A Reinforcement Learning Planner for Irregular Parking Environments](https://mlsanity.com/paper/2608.24485/) (2026)
- [Evaluation Is All You Need for Multi-Modal Autonomous Driving](https://mlsanity.com/paper/2609.30818/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal policy by directly maximizing an objective function using gradient descent methods, in which the objective function is usually the expectation of accumulative future rewards. The latter indirectly finds the optimal policy by solving the Bellman equation, which is the sufficient and necessary condition from Bellman's principle of optimality. We study policy gradient forms of direct and indirect RL and show that both of them can derive the actor-critic architecture and can be unified into a policy gradient with the approximate value function and the stationary state distribution, revealing the equivalence of direct and indirect RL. We employ a Gridworld task to verify the influence of different forms of policy gradient, suggesting their differences and relationships experimentally. Finally, we classify current mainstream RL algorithms using the direct and indirect taxonomy, together with other ones including value-based and policy-based, model-based and model-free.