# Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.04222 · HTML: https://mlsanity.com/paper/2006.04222/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 30 of 38)

- [SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07489/) (2022)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization](https://mlsanity.com/paper/2311.01753/) (2023)
- [Generalization in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2202.00104/) (2022)
- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2404.10976/) (2024)
- [LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.02561/) (2022)
- [Multi-Agent Policy Transfer via Task Relationship Modeling](https://mlsanity.com/paper/2203.04482/) (2022)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2111.05992/) (2021)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation](https://mlsanity.com/paper/2503.21200/) (2025)
- [Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing](https://mlsanity.com/paper/2103.01009/) (2021)
- [Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.12988/) (2021)
- [Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework](https://mlsanity.com/paper/2207.11143/) (2022)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03429/) (2023)
- [S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2206.11054/) (2022)
- [Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2408.13567/) (2024)
- [Closed-Loop Vision-Language Planning for Multi-Agent Coordination](https://mlsanity.com/paper/2502.10148/) (2025)
- [Meta-CPR: Generalize to Unseen Large Number of Agents with Communication Pattern Recognition Module](https://mlsanity.com/paper/2112.07222/) (2021)
- [The StarCraft Multi-Agent Challenges+ : Learning of Multi-Stage Tasks and Environmental Factors without Precise Reward Functions](https://mlsanity.com/paper/2207.02007/) (2022)
- [Transferable Multi-Agent Reinforcement Learning with Dynamic Participating Agents](https://mlsanity.com/paper/2208.02424/) (2022)
- [CORD: Generalizable Cooperation via Role Diversity](https://mlsanity.com/paper/2501.02221/) (2025)
- [GTDE: Grouped Training with Decentralized Execution for Multi-agent Actor-Critic](https://mlsanity.com/paper/2501.10367/) (2024)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.02304/) (2025)
- [Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.10803/) (2022)
- [Hierarchically Structured Scheduling and Execution of Tasks in a Multi-Agent Environment](https://mlsanity.com/paper/2203.03021/) (2022)

## Abstract (arXiv)

Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities. Our method aims to leverage these commonalities by asking the question: ``What is the expected utility of each agent when only considering a randomly selected sub-group of its observed entities?'' By posing this counterfactual question, we can recognize state-action trajectories within sub-groups of entities that we may have encountered in another task and use what we learned in that task to inform our prediction in the current one. We then reconstruct a prediction of the full returns as a combination of factors considering these disjoint groups of entities and train this ``randomly factorized" value function as an auxiliary objective for value-based multi-agent reinforcement learning. By doing so, our model can recognize and leverage similarities across tasks to improve learning efficiency in a multi-task setting. Our approach, Randomized Entity-wise Factorization for Imagined Learning (REFIL), outperforms all strong baselines by a significant margin in challenging multi-task StarCraft micromanagement settings.