# Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning

arXiv: https://arxiv.org/abs/2512.18763 · HTML: https://mlsanity.com/paper/2512.18763/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 71 references that are in the map.
- First posted on arXiv 21 Dec 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Backpropagation-Friendly Eigendecomposition](https://mlsanity.com/paper/1906.09023/) (2019)
- [Eigendecomposition-free Training of Deep Networks with Zero Eigenvalue-based Losses](https://mlsanity.com/paper/1803.08071/) (2018)
- [Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering](https://mlsanity.com/paper/2403.20020/) (2024)
- [Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning](https://mlsanity.com/paper/2210.11317/) (2022)
- [Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering](https://mlsanity.com/paper/2309.07548/) (2023)
- [Neural Network Layers for Prediction of Positive Definite Elastic Stiffness Tensors](https://mlsanity.com/paper/2203.13938/) (2022)
- [Online reinforcement learning via sparse Gaussian mixture model Q-functions](https://mlsanity.com/paper/2509.14585/) (2025)

## Built on by (most cited first, 1 of 1)

- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)

## Abstract (arXiv)

Unlike their conventional use as estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses. The proposed estimators, termed GMM-QFs, possess substantial representational capacity, as they are shown to be universal approximators over a broad class of functions. They are further embedded within Bellman residuals, where their learnable parameters--a fixed number of mixing weights, together with Gaussian mean vectors and covariance matrices--are inferred from data via optimization on a product Riemannian manifold. This geometric perspective on the parameter space naturally introduces Riemannian optimization into the policy-evaluation step of standard policy-iteration (PI) frameworks. Moreover, rigorous theoretical analysis establishes performance bounds on Q-function estimation error under the proposed PI scheme. Supporting numerical tests show that GMM-QFs deliver competitive performance and, in some cases, outperform state-of-the-art approaches across a range of benchmark RL tasks, all while maintaining a significantly smaller computational footprint than deep-learning methods.