mlsanityData to 14 Oct 2026

Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes

arXiv 2608.22765 ↗First posted 24 Aug 2026cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Coupled-dynamics environments expose the one-step outcomes that would follow from several possible counterfactual actions under a common realization of exogenous randomness. The ordinary Markov decision process formalism allows one to reason about the marginal law of each action but discards dependence across these counterfactual outcomes. The Joint Markov decision process (JMDP) formalism preserves that dependence. Prior work established the formalism and solved the fixed-policy joint moment evaluation problem in JMDPs. This paper develops optimal-control methods. We define a nonparametric distributional Bellman optimality operator for JMDPs, and prove that when the induced marginal MDP has a unique optimal policy, its iterates converge in Wasserstein distance to the optimal joint return law. For the first two moments, we establish convergence under a weaker condition that permits several mean-optimal actions as long as their tie resolutions share a second-moment fixed point. We also derive sampled targets for neural approximation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

16 of 27 references.

References · 16

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  5. 2014Algorithms for CVaR Optimization in MDPscited by 99
  6. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  7. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 82
  8. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 22
  9. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 17
  10. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  11. 2021Learning Generalized Gumbel-max Causal Mechanismscited by 16
  12. 2022Counterfactual Analysis in Dynamic Latent State Modelscited by 10
4 more
  1. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  2. 2025Counterfactual Realizabilitycited by 4
  3. 2025Robust Counterfactual Inference in Markov Decision Processescited by 3
  4. 2026Joint MDPs and Reinforcement Learning in Coupled-Dynamics Environmentscited by 1

Built on by

Not cited yet.