mlsanityData to 7 Oct 2026

Foundations of Multivariate Distributional Reinforcement Learning

arXiv 2409.00328 ↗First posted 31 Aug 2024cs.LG math.OC stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), the consideration of multivariate reward signals has led to fundamental advancements in multi-objective decision-making, transfer learning, and representation learning. This work introduces the first oracle-free and computationally-tractable algorithms for provably convergent multivariate distributional dynamic programming and temporal difference learning. Our convergence rates match the familiar rates in the scalar reward setting, and additionally provide new insights into the fidelity of approximate return distribution representations as a function of the reward dimension. Surprisingly, when the reward dimension is larger than 1, we show that standard analysis of categorical TD learning fails, which we resolve with a novel projection onto the space of mass-1 signed measures. Finally, with the aid of our technical results and simulations, we identify tradeoffs between distribution representations that influence the performance of multivariate distributional RL in practice.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 31 Aug 2024.

Builds on

22 of 55 references.

References · 22

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  4. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  5. 2024Reinforcement learningcited by 236
  6. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  7. 2021Efficient and Modular Implicit Differentiationcited by 199
  8. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  9. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  10. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  11. 2021DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learningcited by 26
  12. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
10 more
  1. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  2. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  3. 2019Distributional reinforcement learning with linear function approximationcited by 15
  4. 2024A Distributional Analogue to the Successor Representationcited by 15
  5. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  6. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  7. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 7
  8. 2024Neural-Kernel Conditional Mean Embeddingscited by 7
  9. 2024Off-Policy Reinforcement Learning with High Dimensional Rewardcited by 2
  10. 2021On the Complexity of the Geometric Median Problem with Outlierscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025Temporal Difference Flowscited by 20
  2. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  3. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  4. 2026Compositional Planning with Jumpy World Modelscited by 4
  5. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  6. 2026Joint MDPs and Reinforcement Learning in Coupled-Dynamics Environmentscited by 1
  7. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  8. 2025Distributional Inverse Reinforcement Learningcited by 0
  9. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 0
  10. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discountingcited by 0
  11. 2026Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Drivingcited by 0
  12. 2026Softmax gradient policy for variance minimization and risk-averse multi armed banditscited by 0
4 more
  1. 2026Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimizationcited by 0
  2. 2026Utility-Constrained Policy Optimizationcited by 0
  3. 2026Momba: Network Modernization Improves Multi-Objective Reinforcement Learningcited by 0
  4. 2026Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processescited by 0