# CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zero

arXiv: https://arxiv.org/abs/2505.21536 · HTML: https://mlsanity.com/paper/2505.21536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 91 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [CORL: Research-oriented Deep Offline Reinforcement Learning Library](https://mlsanity.com/paper/2210.07105/) (2022)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [d3rlpy: An Offline Deep Reinforcement Learning Library](https://mlsanity.com/paper/2111.03788/) (2021)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [panda-gym: Open-source goal-conditioned environments for robotic learning](https://mlsanity.com/paper/2106.13687/) (2021)
- [ChainerRL: A Deep Reinforcement Learning Library](https://mlsanity.com/paper/1912.03905/) (2019)
- [skrl: Modular and Flexible Library for Reinforcement Learning](https://mlsanity.com/paper/2202.03825/) (2022)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice](https://mlsanity.com/paper/2406.12905/) (2024)
- [LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery](https://mlsanity.com/paper/2302.09606/) (2023)
- [A Review of Nine Physics Engines for Reinforcement Learning Research](https://mlsanity.com/paper/2407.08590/) (2024)
- [Thermodynamical Material Networks for Modeling, Planning, and Control of Circular Material Flows](https://mlsanity.com/paper/2111.10693/) (2021)
- [A Comparison of Reinforcement Learning Frameworks for Software Testing Tasks](https://mlsanity.com/paper/2208.12136/) (2022)
- [Circularity of Thermodynamical Material Networks: Indicators, Examples, and Algorithms](https://mlsanity.com/paper/2209.15051/) (2022)
- [A Unification Between Deep-Learning Vision, Compartmental Dynamical Thermodynamics, and Robotic Manipulation for a Circular Economy](https://mlsanity.com/paper/2405.14406/) (2024)
- [Deep multi-intentional inverse reinforcement learning for cognitive multi-function radar inverse cognition](https://mlsanity.com/paper/2408.08478/) (2024)
- [Synchronized Object Detection for Autonomous Sorting, Mapping, and Quantification of Materials in Circular Healthcare](https://mlsanity.com/paper/2405.06821/) (2024)
- [Circular Microalgae-Based Carbon Control for Net Zero](https://mlsanity.com/paper/2502.02382/) (2025)

## Abstract (arXiv)

The demand of finite raw materials will keep increasing as they fuel modern society. Simultaneously, solutions for stopping carbon emissions in the short term are not available, thus making the net zero target extremely challenging to achieve at scale. The circular economy (CE) paradigm is gaining attention as a solution to address climate change and the uncertainties of supplies of critical materials. Hence, in this paper, we introduce CiRL, a deep reinforcement learning (DRL) library of environments focused on the circularity control of both solid and fluid materials. The integration of DRL into the design of material circularity is possible thanks to the formalism of thermodynamical material networks, which is underpinned by compartmental dynamical thermodynamics. Along with the focus on circularity, this library has three more features: the new CE-oriented environments are in the state-space form, which is typically used in dynamical systems analysis and control design; it is based on a state-of-the-art Python library of DRL algorithms, namely, Stable-Baselines3; and it is developed in Google Colaboratory to be accessible to researchers from different disciplines and backgrounds as is often the case for circular economy researchers and engineers. CiRL is intended to be a tool to generate AI-driven actions for optimizing the circularity of supply-recovery chains and to be combined with human-driven decisions derived from material flow analysis (MFA) studies. CiRL is publicly available.