# PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice

arXiv: https://arxiv.org/abs/2406.12905 · HTML: https://mlsanity.com/paper/2406.12905/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine](https://mlsanity.com/paper/2206.10558/) (2022)
- [The Neural MMO Platform for Massively Multiagent Research](https://mlsanity.com/paper/2110.07594/) (2021)

## Built on by (most cited first, 24 of 24)

- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS](https://mlsanity.com/paper/2408.01584/) (2024)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [Human-like autonomy emerges from self-play and a pinch of human data](https://mlsanity.com/paper/2606.19370/) (2026)
- [Scaling Self-Play for End-to-End Driving](https://mlsanity.com/paper/2606.19641/) (2026)
- [TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations](https://mlsanity.com/paper/2606.17386/) (2026)
- [Syllabus: Portable Curricula for Reinforcement Learning Agents](https://mlsanity.com/paper/2411.11318/) (2024)
- [Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning](https://mlsanity.com/paper/2507.07197/) (2025)
- [CAMAR: Continuous Actions Multi-Agent Routing](https://mlsanity.com/paper/2508.12845/) (2025)
- [Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale](https://mlsanity.com/paper/2511.04904/) (2025)
- [Equivariant Reinforcement Learning for Clifford Quantum Circuit Synthesis](https://mlsanity.com/paper/2605.10910/) (2026)
- [Pictura: Perspective-View Self-Play at Scale for Driving](https://mlsanity.com/paper/2607.26005/) (2026)
- [Pokemon Red via Reinforcement Learning](https://mlsanity.com/paper/2502.19920/) (2025)
- [The challenge of hidden gifts in multi-agent reinforcement learning](https://mlsanity.com/paper/2505.20579/) (2025)
- [TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale](https://mlsanity.com/paper/2607.13028/) (2026)
- [CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zero](https://mlsanity.com/paper/2505.21536/) (2025)
- [Assistax: A Multi-Agent Hardware-Accelerated Reinforcement Learning Benchmark for Assistive Robotics](https://mlsanity.com/paper/2507.21638/) (2025)
- [Automatic Generation of High-Performance RL Environments](https://mlsanity.com/paper/2603.12145/) (2026)
- [A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations](https://mlsanity.com/paper/2604.27162/) (2026)
- [PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games](https://mlsanity.com/paper/2609.09059/) (2026)
- [Comfort by Construction: Adaptive, Comfort-Bounded Action Spaces for Learned Driving Policies](https://mlsanity.com/paper/2609.13011/) (2026)
- [VISTA: An Attention-Based Multi-Agent Reinforcement Learning Architecture for Space Situational Awareness Sensor Tasking](https://mlsanity.com/paper/2609.23875/) (2026)
- [Learning in Dreams, Winning in Reality: A Continuous Dyna Loop for a Ten-Hero MOBA](https://mlsanity.com/paper/2610.08033/) (2026)
- [Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies](https://mlsanity.com/paper/2610.19714/) (2026)

## Abstract (arXiv)

You have an environment, a model, and a reinforcement learning library that are designed to work together but don't. PufferLib makes them play nice. The library provides one-line environment wrappers that eliminate common compatibility problems and fast vectorization to accelerate training. With PufferLib, you can use familiar libraries like CleanRL and SB3 to scale from classic benchmarks like Atari and Procgen to complex simulators like NetHack and Neural MMO. We release pip packages and prebuilt images with dependencies for dozens of environments. All of our code is free and open-source software under the MIT license, complete with baselines, documentation, and support at pufferai.github.io.