# Thermodynamics of Reinforcement Learning Curricula

arXiv: https://arxiv.org/abs/2603.12324 · HTML: https://mlsanity.com/paper/2603.12324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Quantifying Differences in Reward Functions](https://mlsanity.com/paper/2006.13900/) (2020)
- [Fluctuation-dissipation relations for stochastic gradient descent](https://mlsanity.com/paper/1810.00004/) (2018)
- [The Value Function Polytope in Reinforcement Learning](https://mlsanity.com/paper/1901.11524/) (2019)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Dynamics-Aware Comparison of Learned Reward Functions](https://mlsanity.com/paper/2201.10081/) (2022)
- [Reinforcement learning of rare diffusive dynamics](https://mlsanity.com/paper/2105.04321/) (2021)
- [RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2408.01972/) (2024)
- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [Entropy Regularized Reinforcement Learning Using Large Deviation Theory](https://mlsanity.com/paper/2106.03931/) (2021)
- [A Differential Perspective on Distributional Reinforcement Learning](https://mlsanity.com/paper/2506.03333/) (2025)
- [Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations](https://mlsanity.com/paper/2209.14089/) (2022)

## Abstract (arXiv)

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.