mlsanityData to 14 Oct 2026

Joint MDPs and Reinforcement Learning in Coupled-Dynamics Environments

arXiv 2603.06946 ↗First posted 6 Mar 2026cs.LG math.OCcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Many distributional quantities in reinforcement learning are intrinsically joint across actions, including distributions of gaps and probabilities of superiority. However, the classical Markov decision process (MDP) formalism specifies only marginal laws and leaves the joint law of counterfactual one-step outcomes across multiple possible actions at a state unspecified. We study coupled-dynamics environments with a multi-action generative interface which can sample counterfactual one-step outcomes for multiple actions under shared exogenous randomness. We propose joint MDPs (JMDPs) as a formalism for such environments by augmenting an MDP with a multi-action sample transition model which specifies a coupling of one-step counterfactual outcomes, while preserving standard MDP interaction as marginal observations. We adopt and formalize a one-step coupling regime where dependence across actions is confined to immediate counterfactual outcomes at the queried state. In this regime, we derive Bellman operators for nth-order return moments, providing dynamic programming and incremental algorithms with convergence guarantees.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 37 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

13 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning14 Oct
  2. 2024Foundations of Multivariate Distributional Reinforcement Learning31 Aug

Other references · 11

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2014Algorithms for CVaR Optimization in MDPscited by 99
  5. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  6. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 59
  7. 2013PEGASUS: A Policy Search Method for Large MDPs and POMDPscited by 22
  8. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 22
  9. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  10. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
  11. 2020Solving High-Order Portfolios via Successive Convex Approximation Algorithmscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning in Markov Decision Processes with Exogenous Dynamics
  2. 2026Mathematical methods of reinforcement learning
  3. 2024Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning
  4. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
  5. 2025Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning
  6. 2021Reinforcement Learning in Reward-Mixing MDPs
  7. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  8. 2021Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity