mlsanityData to 7 Oct 2026

Podracer architectures for scalable Reinforcement Learning

arXiv 2104.06272 ↗First posted 13 Apr 2021cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the ability to deploy experiments at a scale traditionally associated with production systems.Deep learning frameworks such as TensorFlow, PyTorch and JAX allow users to transparently make use of accelerators, such as TPUs and GPUs, to offload the more computationally intensive parts of training and inference in modern deep learning systems. Popular training pipelines that use these frameworks for deep learning typically focus on (un-)supervised learning. How to best train reinforcement learning (RL) agents at scale is still an active research area. In this report we argue that TPUs are particularly well suited for training RL agents in a scalable, efficient and reproducible way. Specifically we describe two architectures designed to make the best use of the resources available on a TPU Pod (a special configuration in a Google data center that features multiple TPU devices connected to each other by extremely low latency communication channels).

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 12 of its 36 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

12 of 36 references.

References · 12

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  7. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  8. 2018Distributed Prioritized Experience Replaycited by 260
  9. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  10. 2020Discovering Reinforcement Learning Algorithmscited by 75
  11. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  12. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 10; 2024: 3; 2025: 7; 2026: 3.

37 papers.

Sort:

By area:

  1. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  2. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  3. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  4. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  5. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 52
  6. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
  7. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  8. 2022Melting Pot 2.0cited by 37
  9. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  10. 2021Proper Value Equivalencecited by 33
  11. 2022The Phenomenon of Policy Churncited by 28
  12. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
25 more
  1. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 20
  2. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  3. 2021Mava: a research library for distributed multi-agent reinforcement learning in JAXcited by 14
  4. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  5. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  6. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  7. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  8. 2023SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Corescited by 10
  9. 2021ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learningcited by 9
  10. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  11. 2021Self-Consistent Models and Valuescited by 7
  12. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  13. 2023Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platformcited by 7
  14. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  15. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  16. 2022MSRL: Distributed Reinforcement Learning with Dataflow Fragmentscited by 4
  17. 2025Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscalecited by 2
  18. 2023marl-jax: Multi-Agent Reinforcement Leaning Frameworkcited by 1
  19. 2025Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphscited by 1
  20. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  21. 2024Enabling High Data Throughput Reinforcement Learning on GPUs: A Domain Agnostic Framework for Data-Driven Scientific Researchcited by 0
  22. 2025Chargax: A JAX Accelerated EV Charging Simulatorcited by 0
  23. 2026JaxARC: A High-Performance JAX-based Environment for Abstraction and Reasoning Researchcited by 0
  24. 2026Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environmentcited by 0
  25. 2026Delightful Distributed Policy Gradientcited by 0