mlsanityData to 7 Oct 2026

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

arXiv 1802.01561 ↗First posted 5 Feb 2018cs.LG cs.AIcited by 866

Not on the map. Why?

Cited by 63 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

Key facts. Cited by 866 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 1; 2020: 4; 2021: 188; 2022: 156; 2023: 147; 2024: 105; 2025: 131; 2026: 130.

866 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  5. 2022A Generalist Agentcited by 802
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2021Domain Generalization with MixStylecited by 563
  8. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
38 more
  1. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  5. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  6. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  7. 2022Multi-Game Decision Transformerscited by 212
  8. 2021The Values Encoded in Machine Learning Researchcited by 172
  9. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  10. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  11. 2025Flow Q-Learningcited by 162
  12. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  13. 2021A System for General In-Hand Object Re-Orientationcited by 155
  14. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  15. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  16. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  17. 2021Collaborating with Humans without Human Datacited by 139
  18. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  19. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  20. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  21. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  22. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  23. 2023Learning to Act without Actionscited by 116
  24. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  25. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  26. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  27. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  28. 2022Discovered Policy Optimisationcited by 103
  29. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  30. 2023Synthetic Experience Replaycited by 100
  31. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  32. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  33. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  34. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  35. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  36. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  37. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  38. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86

50 most cited of 866 All 866 (JSON)