# Online reinforcement learning via sparse Gaussian mixture model Q-functions

arXiv: https://arxiv.org/abs/2509.14585 · HTML: https://mlsanity.com/paper/2509.14585/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization](https://mlsanity.com/paper/2307.03571/) (2023)
- [Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering](https://mlsanity.com/paper/2403.20020/) (2024)
- [Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning](https://mlsanity.com/paper/2210.11317/) (2022)
- [Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering](https://mlsanity.com/paper/2309.07548/) (2023)

## Built on by (most cited first, 2 of 2)

- [Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning](https://mlsanity.com/paper/2512.18763/) (2025)
- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)

## Abstract (arXiv)

This paper introduces a structured and interpretable online policy-iteration framework for reinforcement learning (RL), built around the novel class of sparse Gaussian mixture model Q-functions (S-GMM-QFs). Extending earlier work that trained GMM-QFs offline, the proposed framework develops an online scheme that leverages streaming data to encourage exploration. Model complexity is regulated through sparsification by Hadamard overparametrization, which mitigates overfitting while preserving expressiveness. The parameter space of S-GMM-QFs is naturally endowed with a Riemannian manifold structure, allowing for principled parameter updates via online gradient descent on a smooth objective. Numerical experiments show that S-GMM-QFs match or even outperform dense deep RL (DeepRL) methods on standard benchmarks while using significantly fewer parameters. Moreover, they maintain strong performance even in low-parameter regimes where sparsified DeepRL methods fail to generalize.