# Tianshou: a Highly Modularized Deep Reinforcement Learning Library

arXiv: https://arxiv.org/abs/2107.14171 · HTML: https://mlsanity.com/paper/2107.14171/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Learning values across many orders of magnitude](https://mlsanity.com/paper/1602.07714/) (2016)
- [Time Limits in Reinforcement Learning](https://mlsanity.com/paper/1712.00378/) (2017)
- [ChainerRL: A Deep Reinforcement Learning Library](https://mlsanity.com/paper/1912.03905/) (2019)
- [MushroomRL: Simplifying Reinforcement Learning Research](https://mlsanity.com/paper/2001.01102/) (2020)
- [Multi-step Reinforcement Learning: A Unifying Algorithm](https://mlsanity.com/paper/1703.01327/) (2017)
- [Accelerated Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1803.02811/) (2018)
- [rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch](https://mlsanity.com/paper/1909.01500/) (2019)
- [Efficient Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1705.04862/) (2017)

## Built on by (most cited first, 30 of 143)

- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [CORL: Research-oriented Deep Offline Reinforcement Learning Library](https://mlsanity.com/paper/2210.07105/) (2022)
- [d3rlpy: An Offline Deep Reinforcement Learning Library](https://mlsanity.com/paper/2111.03788/) (2021)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark](https://mlsanity.com/paper/2306.17100/) (2023)
- [EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine](https://mlsanity.com/paper/2206.10558/) (2022)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Datasets and Benchmarks for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2306.09303/) (2023)
- [Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learning](https://mlsanity.com/paper/2303.17503/) (2023)
- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [Laminar: A Scalable Asynchronous RL Post-Training Framework](https://mlsanity.com/paper/2510.12633/) (2025)
- [Multi-agent Dynamic Algorithm Configuration](https://mlsanity.com/paper/2210.06835/) (2022)
- [Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo](https://mlsanity.com/paper/2305.18246/) (2023)
- [Weight Clipping for Deep Continual and Reinforcement Learning](https://mlsanity.com/paper/2407.01704/) (2024)
- [Effective control of two-dimensional Rayleigh--B'enard convection: invariant multi-agent reinforcement learning is all you need](https://mlsanity.com/paper/2304.02370/) (2023)
- [EgoGen: An Egocentric Synthetic Data Generator](https://mlsanity.com/paper/2401.08739/) (2024)
- [SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning](https://mlsanity.com/paper/2403.09110/) (2024)
- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice](https://mlsanity.com/paper/2406.12905/) (2024)
- [Recurrent Off-policy Baselines for Memory-based Continuous Control](https://mlsanity.com/paper/2110.12628/) (2021)
- [MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library](https://mlsanity.com/paper/2210.13708/) (2022)
- [Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning](https://mlsanity.com/paper/2402.03046/) (2024)
- [Learning Shared Safety Constraints from Multi-task Demonstrations](https://mlsanity.com/paper/2309.00711/) (2023)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)

## Abstract (arXiv)

In this paper, we present Tianshou, a highly modularized Python library for deep reinforcement learning (DRL) that uses PyTorch as its backend. Tianshou intends to be research-friendly by providing a flexible and reliable infrastructure of DRL algorithms. It supports online and offline training with more than 20 classic algorithms through a unified interface. To facilitate related research and prove Tianshou's reliability, we have released Tianshou's benchmark of MuJoCo environments, covering eight classic algorithms with state-of-the-art performance. We open-sourced Tianshou at https://github.com/thu-ml/tianshou/.