# Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies

arXiv: https://arxiv.org/abs/2405.01198 · HTML: https://mlsanity.com/paper/2405.01198/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 2 May 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Leveraging exploration in off-policy algorithms via normalizing flows](https://mlsanity.com/paper/1905.06893/) (2019)
- [Safe Reinforcement Learning via Probabilistic Logic Shields](https://mlsanity.com/paper/2303.03226/) (2023)
- [Discrete and Continuous Action Representation for Practical RL in Video Games](https://mlsanity.com/paper/1912.11077/) (2019)
- [Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies](https://mlsanity.com/paper/1906.02771/) (2019)
- [Verifiably Safe Exploration for End-to-End Reinforcement Learning](https://mlsanity.com/paper/2007.01223/) (2020)
- [FlowPG: Action-constrained Policy Gradient with Normalizing Flows](https://mlsanity.com/paper/2402.05149/) (2024)
- [Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning](https://mlsanity.com/paper/2311.15341/) (2023)
- [Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning](https://mlsanity.com/paper/2310.02360/) (2023)

## Built on by (most cited first, 1 of 1)

- [Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation](https://mlsanity.com/paper/2606.31043/) (2026)

## Abstract (arXiv)

Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions.