mlsanityData to 7 Oct 2026

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning

arXiv 2402.08421 ↗First posted 13 Feb 2024cs.LG cs.MAcited by 10

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct access to the physical environment. This limitation is particularly severe in multi-agent systems, for which conventional multi-agent reinforcement learning (MARL) requires a large number of coordinated online interactions with the environment during training. When only offline data is available, a direct application of online MARL schemes would generally fail due to the epistemic uncertainty entailed by the lack of exploration during training. In this work, we propose an offline MARL scheme that integrates distributional RL and conservative Q-learning to address the environment's inherent aleatoric uncertainty and the epistemic uncertainty arising from the use of offline data. We explore both independent and joint learning strategies. The proposed MARL scheme, referred to as multi-agent conservative quantile regression, addresses general risk-sensitive design criteria and is applied to the trajectory planning problem in drone networks, showcasing its advantages.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 56 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

37 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets19 Nov
  2. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  3. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem26 May
  4. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  5. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 32

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  11. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  12. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 163
20 more
  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  2. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  3. 2018Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networkscited by 70
  4. 2021Simulation Intelligence: Towards a New Generation of Scientific Methodscited by 68
  5. 2018Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networkscited by 54
  6. 2020Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learningcited by 41
  7. 2021Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directionscited by 38
  8. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  9. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  10. 2021Imitation Learning by Reinforcement Learningcited by 17
  11. 2019Deep Reinforcement Learning for Minimizing Age-of-Information in UAV-assisted Networkscited by 13
  12. 2023Multi-UAV Path Learning for Age and Power Optimization in IoT with UAV Battery Rechargecited by 9
  13. 2022Traffic Prediction and Fast Uplink for Hidden Markov IoT Modelscited by 8
  14. 2023Colosseum as a Digital Twin: Bridging Real-World Experimentation and Wireless Network Emulationcited by 6
  15. 2023Age Minimization in Massive IoT via UAV Swarm: A Multi-agent Reinforcement Learning Approachcited by 5
  16. 2020Millimeter Wave Communications with an Intelligent Reflector: Performance Optimization and Distributional Reinforcement Learningcited by 3
  17. 2022A Learning-Based Trajectory Planning of Multiple UAVs for AoI Minimization in IoT Networkscited by 3
  18. 2024Traffic Learning and Proactive UAV Trajectory Planning for Data Uplink in Markovian IoT Modelscited by 3
  19. 2020Distributional Reinforcement Learning for mmWave Communications with Intelligent Reflectors on a UAVcited by 2
  20. 2023Network Digital Twin for Open RAN: The Key Enablers, Standardization, and Use Casescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2024Offline and Distributional Reinforcement Learning for Radio Resource Managementcited by 3
  2. 2025Offline and Distributional Reinforcement Learning for Wireless Communicationscited by 3
  3. 2024Semantic Meta-Split Learning: A TinyML Scheme for Few-Shot Wireless Image Classificationcited by 1
  4. 2025Resilient UAV Trajectory Planning via Few-Shot Meta-Offline Reinforcement Learningcited by 1
  5. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  6. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 0
  7. 2025ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithmcited by 0
  8. 2026Offline Multi-Agent Reinforcement Learning for 6G Communications: Fundamentals, Applications and Future Directionscited by 0
  9. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 0
  10. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Meta-Offline and Distributional Multi-Agent RL for Risk-Aware Decision-Making ↗
  2. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  3. 2026Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control
  4. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARL
  5. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  6. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
  7. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  8. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning