mlsanityData to 14 Oct 2026

ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithm

arXiv 2510.15006 ↗First posted 16 Oct 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In most value-based reinforcement learning (RL) algorithms, the agent estimates only the expected reward for each action and selects the action with the highest reward. In contrast, Distributional Reinforcement Learning (DRL) estimates the entire probability distribution of possible rewards, providing richer information about uncertainty and variability. C51 is a popular DRL algorithm for discrete action spaces. It uses a Q-learning approach, where the distribution is learned using a greedy Bellman update. However, this can cause problems if multiple actions at a state have similar expected reward but with different distributions, as the algorithm may not learn a stable distribution. This study presents a modified version of C51 (ES-C51) that replaces the greedy Q-learning update with an Expected Sarsa update, which uses a softmax calculation to combine information from all possible actions at a state rather than relying on a single best action. This reduces instability when actions have similar expected rewards and allows the agent to learn higher-performing policies. This approach is evaluated on classic control environments from Gym, and Atari-10 games. For a fair comparison, we modify the standard C51's exploration strategy from e-greedy to softmax, which we refer to as QL-C51 (Q- Learning based C51). The results demonstrate that ES-C51 outperforms QL-C51 across many environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 23 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

14 of 23 references.

References · 14

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  5. 2020Revisiting Fundamentals of Experience Replaycited by 175
  6. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  7. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  8. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  9. 2022The Phenomenon of Policy Churncited by 28
  10. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  11. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  12. 2022Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learningcited by 5
2 more
  1. 2024Revisiting Experience Replayable Conditionscited by 3
  2. 2025Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regressioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Distributional Soft Actor-Critic with Diffusion Policy
  2. 2026Distributional Reinforcement Learning via the Cram'er Distance
  3. 2025A Differential Perspective on Distributional Reinforcement Learning
  4. 2024Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)
  5. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  6. 2021Conservative Offline Distributional Reinforcement Learning
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning
  8. 2024On Policy Evaluation Algorithms in Distributional Reinforcement Learning