# TorchRL: A data-driven decision-making library for PyTorch

arXiv: https://arxiv.org/abs/2306.00577 · HTML: https://mlsanity.com/paper/2306.00577/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 105 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 75)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)

## Built on by (most cited first, 30 of 60)

- [RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark](https://mlsanity.com/paper/2306.17100/) (2023)
- [Generative Flow Networks as Entropy-Regularized RL](https://mlsanity.com/paper/2310.12934/) (2023)
- [Laminar: A Scalable Asynchronous RL Post-Training Framework](https://mlsanity.com/paper/2510.12633/) (2025)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [BenchMARL: Benchmarking Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.01472/) (2023)
- [No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO](https://mlsanity.com/paper/2405.00662/) (2024)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [GEM: A Gym for Agentic LLMs](https://mlsanity.com/paper/2510.01051/) (2025)
- [Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning](https://mlsanity.com/paper/2402.03046/) (2024)
- [ACEGEN: Reinforcement learning of generative chemical agents for drug discovery](https://mlsanity.com/paper/2405.04657/) (2024)
- [Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.15054/) (2024)
- [CREW: Facilitating Human-AI Teaming Research](https://mlsanity.com/paper/2408.00170/) (2024)
- [IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL](https://mlsanity.com/paper/2306.11551/) (2023)
- [Pearl: A Production-ready Reinforcement Learning Agent](https://mlsanity.com/paper/2312.03814/) (2023)
- [AFABench: A Generic Framework for Benchmarking Active Feature Acquisition](https://mlsanity.com/paper/2508.14734/) (2025)
- [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://mlsanity.com/paper/2508.00234/) (2025)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Adaptive Destruction Processes for Diffusion Samplers](https://mlsanity.com/paper/2506.01541/) (2025)
- [Reinforcement learning with combinatorial actions for coupled restless bandits](https://mlsanity.com/paper/2503.01919/) (2025)
- [RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control](https://mlsanity.com/paper/2306.03530/) (2023)
- [In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2407.16807/) (2024)
- [The impact of behavioral diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2412.16244/) (2024)
- [URB -- Urban Routing Benchmark for RL-equipped Connected Autonomous Vehicles](https://mlsanity.com/paper/2505.17734/) (2025)
- [ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments](https://mlsanity.com/paper/2609.19134/) (2026)
- [Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling](https://mlsanity.com/paper/2410.04988/) (2024)
- [RouteRL: Multi-agent reinforcement learning framework for urban route choice with autonomous vehicles](https://mlsanity.com/paper/2502.20065/) (2025)
- [Scalable Production Scheduling: Linear Complexity via Unified Homogeneous Graphs](https://mlsanity.com/paper/2604.23841/) (2026)
- [When Is Diversity Rewarded in Cooperative Multi-Agent Learning?](https://mlsanity.com/paper/2506.09434/) (2025)
- [Reinforcement Learning Increases Wind Farm Power Production by Enabling Closed-Loop Collaborative Control](https://mlsanity.com/paper/2506.20554/) (2025)
- [Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control](https://mlsanity.com/paper/2601.15015/) (2026)

## Abstract (arXiv)

PyTorch has ascended as a premier machine learning framework, yet it lacks a native and comprehensive library for decision and control tasks suitable for large development teams dealing with complex real-world data and environments. To address this issue, we propose TorchRL, a generalistic control library for PyTorch that provides well-integrated, yet standalone components. We introduce a new and flexible PyTorch primitive, the TensorDict, which facilitates streamlined algorithm development across the many branches of Reinforcement Learning (RL) and control. We provide a detailed description of the building blocks and an extensive overview of the library across domains and tasks. Finally, we experimentally demonstrate its reliability and flexibility and show comparative benchmarks to demonstrate its computational efficiency. TorchRL fosters long-term support and is publicly available on GitHub for greater reproducibility and collaboration within the research community. The code is open-sourced on GitHub.