mlsanityData to 7 Oct 2026

Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning

arXiv 2304.00977 ↗First posted 31 Mar 2023cs.AI cs.LG cs.MAcited by 2

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

'Reincarnation' in reinforcement learning has been proposed as a formalisation of reusing prior computation from past experiments when training an agent in an environment. In this paper, we present a brief foray into the paradigm of reincarnation in the multi-agent (MA) context. We consider the case where only some agents are reincarnated, whereas the others are trained from scratch -- selective reincarnation. In the fully-cooperative MA setting with heterogeneous agents, we demonstrate that selective reincarnation can lead to higher returns than training fully from scratch, and faster convergence than training with full reincarnation. However, the choice of which agents to reincarnate in a heterogeneous system is vitally important to the outcome of the training -- in fact, a poor choice can lead to considerably worse results than the alternatives. We argue that a rich field of work exists here, and we hope that our effort catalyses further energy in bringing the topic of reincarnation to the multi-agent realm.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 43 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

27 of 43 references.

References · 27

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2019The StarCraft Multi-Agent Challengecited by 589
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  12. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
15 more
  1. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  2. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  3. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  4. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  5. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  6. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 90
  7. 2018Kickstarting Deep Reinforcement Learningcited by 75
  8. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  9. 2020Agent Modelling under Partial Observability for Deep Reinforcement Learningcited by 56
  10. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  11. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  12. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 31
  13. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  14. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  15. 2021KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  2. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification
  2. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
  3. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
  4. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
  5. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
  6. 2026Recursive Agent Optimization
  7. 2018PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
  8. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents