mlsanityData to 7 Oct 2026

Ergodicity in reinforcement learning

arXiv 2603.10895 ↗First posted 11 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e., the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 52 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

17 of 52 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  5. 2013Risk-sensitive Reinforcement Learningcited by 129
  6. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
  7. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  8. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  9. 2018Exploration in Structured Reinforcement Learningcited by 34
  10. 2018Relative Entropy Regularized Policy Iterationcited by 32
  11. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  12. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
5 more
  1. 2025Rethinking the Foundations for Continual Reinforcement Learningcited by 15
  2. 2019A Learnable Safety Measurecited by 7
  3. 2023Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformationscited by 3
  4. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  5. 2025Revisiting Value Iteration: Unified Analysis of Discounted and Average-Reward Casescited by 1

Built on by

Not cited yet.