mlsanityData to 7 Oct 2026

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model

arXiv 2402.07598 ↗First posted 12 Feb 2024cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023). Our analysis provides new theoretical results on categorical approaches to distributional RL, and also introduces a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. We also provide an experimental study comparing several model-based distributional RL algorithms, with several takeaways for practitioners.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 65 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

24 of 65 references.

References · 24

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  4. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  5. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  6. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  7. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  8. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  9. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  10. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  11. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  12. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
12 more
  1. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  2. 2022Nystr"om Kernel Mean Embeddingscited by 21
  3. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  4. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  5. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  6. 2020Implicit Distributional Reinforcement Learningcited by 16
  7. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  8. 2020Sequential Transfer in Reinforcement Learning with a Generative Modelcited by 10
  9. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8
  10. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  11. 2018GAN Q-learningcited by 7
  12. 2022Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approachcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  2. 2024Multiple Greedy Quasi-Newton Methods for Saddle Point Problemscited by 3
  3. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  4. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 0
  5. 2026Distributional Off-Policy Evaluation with Deep Quantile Process Regressioncited by 0
  6. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0
  7. 2026Tail-Influence Sampling for CVaR Policy Evaluationcited by 0