# Randomized Value Functions via Multiplicative Normalizing Flows

arXiv: https://arxiv.org/abs/1806.02315 · HTML: https://mlsanity.com/paper/1806.02315/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2018.

## Built on by (most cited first, 15 of 15)

- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Bayesian Bellman Operators](https://mlsanity.com/paper/2106.05012/) (2021)
- [Flow-based Recurrent Belief State Learning for POMDPs](https://mlsanity.com/paper/2205.11051/) (2022)
- [TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed](https://mlsanity.com/paper/2203.10335/) (2022)
- [CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/2210.09496/) (2022)
- [Deep Exploration with PAC-Bayes](https://mlsanity.com/paper/2402.03055/) (2024)
- [Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise](https://mlsanity.com/paper/2203.01004/) (2022)
- [Bayesian Exploration Networks](https://mlsanity.com/paper/2308.13049/) (2023)
- [Learning Uncertainty-Aware Temporally-Extended Actions](https://mlsanity.com/paper/2402.05439/) (2024)
- [Trust-MARL: Trust-Based Multi-Agent Reinforcement Learning Framework for Cooperative On-Ramp Merging Control in Heterogeneous Traffic Flow](https://mlsanity.com/paper/2506.12600/) (2025)
- [Constraining cosmological parameters from N-body simulations with Variational Bayesian Neural Networks](https://mlsanity.com/paper/2301.03991/) (2023)
- [Out-of-Distribution Detection of Melanoma using Normalizing Flows](https://mlsanity.com/paper/2103.12672/) (2021)
- [Exploring More When It Needs in Deep Reinforcement Learning](https://mlsanity.com/paper/2109.13477/) (2021)

## Abstract (arXiv)

Randomized value functions offer a promising approach towards the challenge of efficient exploration in complex environments with high dimensional state and action spaces. Unlike traditional point estimate methods, randomized value functions maintain a posterior distribution over action-space values. This prevents the agent's behavior policy from prematurely exploiting early estimates and falling into local optima. In this work, we leverage recent advances in variational Bayesian neural networks and combine these with traditional Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG) to achieve randomized value functions for high-dimensional domains. In particular, we augment DQN and DDPG with multiplicative normalizing flows in order to track a rich approximate posterior distribution over the parameters of the value function. This allows the agent to perform approximate Thompson sampling in a computationally efficient manner via stochastic gradient methods. We demonstrate the benefits of our approach through an empirical comparison in high dimensional environments.