mlsanityData to 14 Oct 2026

Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

arXiv 2607.23474 ↗First posted 26 Jul 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper develops an online, off-policy policy-iteration framework for reinforcement learning (RL), based on sparse Gaussian-mixture-model Q-functions (S-GMM-QFs). The framework reconciles streaming, non-stationary data with the Riemannian structure of the parameter space while handling distributional mismatch through experience replay. S-GMM-QFs are introduced via Hadamard overparametrization, enabling interpretable sparsification through smooth regularization that facilitates Riemannian-based optimization. Overparametrization allows the framework to adaptively identify meaningful components from a large initial pool, yielding sparse models where interpretability emerges naturally from geometry: each component's parameters (means and covariances) explicitly encode its geometric role in the ambient state-action space. These geometric roles are learned through online gradient descent on a smooth objective over a (Cartesian-product) Riemannian manifold. Numerical tests demonstrate that S-GMM-QFs match or exceed deep RL methods while using substantially fewer parameters and achieving faster improvement per observed transition. Notably, parameter efficiency and interpretability combine to maintain strong generalization in low-parameter regimes where sparsified deep RL approaches degrade.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 55 references that are in the map. First posted on arXiv 26 Jul 2026.

Builds on

22 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning21 Dec
  2. 2025Online reinforcement learning via sparse Gaussian mixture model Q-functions18 Sep
  3. 2024Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering29 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  10. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  11. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  12. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
7 more
  1. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  2. 2018Riemannian Adaptive Optimization Methodscited by 194
  3. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  4. 2023Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrizationcited by 11
  5. 2022Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learningcited by 7
  6. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
  7. 2023Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filteringcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization
  2. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
  3. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
  4. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes
  5. 2024Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning
  6. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
  7. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  8. 2026Residuals-based Offline Reinforcement Learning