mlsanityData to 7 Oct 2026

Weakly Coupled Deep Q-Networks

arXiv 2310.18803 ↗First posted 28 Oct 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We propose weakly coupled deep Q-networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN "subagents", one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems, 310 total actions, and a continuous state space.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 26 of its 64 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

26 of 64 references.

References · 26

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2015Prioritized Experience Replaycited by 1,070
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2015Contextual Markov Decision Processescited by 213
  9. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  10. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  11. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  12. 2020Whittle index based Q-learning for restless bandits with average rewardcited by 43
14 more
  1. 2014Near-optimal Reinforcement Learning in Factored MDPscited by 41
  2. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  3. 2021NeurWIN: Neural Whittle Index Network For Restless Bandits Via Deep RLcited by 27
  4. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  5. 2017An Asymptotically Optimal Index Policy for Finite-Horizon Restless Banditscited by 20
  6. 2018Age of Information: Whittle Index for Scheduling Stochastic Arrivalscited by 20
  7. 2021Q-Learning Lagrange Policies for Multi-Action Restless Banditscited by 20
  8. 2013Hierarchical Solution of Markov Decision Processes using Macro-actionscited by 19
  9. 2016Deadline Scheduling as Restless Banditscited by 16
  10. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  11. 2016On the Whittle Index for Restless Multi-armed Hidden Markov Banditscited by 12
  12. 2022Near-optimality for infinite-horizon restless bandits with many armscited by 9
  13. 2014An Approximate Dynamic Programming Algorithm for Monotone Value Functionscited by 8
  14. 2020Lookahead-Bounded Q-Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025Projection-based Lyapunov method for fully heterogeneous weakly-coupled MDPscited by 2
  2. 2024Deep reinforcement learning for weakly coupled MDP's with continuous actionscited by 1
  3. 2024Fair Resource Allocation in Weakly Coupled Markov Decision Processescited by 0
  4. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processescited by 0
  5. 2025Multi-agent Markov Entanglementcited by 0
  6. 2026Fair Policy Optimization in Major-Minor Weakly Coupled Markov Decision Processescited by 0