# Distributional reinforcement learning with linear function approximation

arXiv: https://arxiv.org/abs/1902.03149 · HTML: https://mlsanity.com/paper/1902.03149/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2019.

## Built on by (most cited first, 16 of 16)

- [An Optimistic Perspective on Offline Reinforcement Learning](https://mlsanity.com/paper/1907.04543/) (2019)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [GMAC: A Distributional Perspective on Actor-Critic Framework](https://mlsanity.com/paper/2105.11366/) (2021)
- [Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes](https://mlsanity.com/paper/2403.12946/) (2024)
- [One-Step Distributional Reinforcement Learning](https://mlsanity.com/paper/2304.14421/) (2023)
- [A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.14172/) (2025)
- [Robustness and risk management via distributional dynamic programming](https://mlsanity.com/paper/2112.15430/) (2021)
- [Value-Distributional Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.06590/) (2023)
- [Conjugated Discrete Distributions for Distributional Reinforcement Learning](https://mlsanity.com/paper/2112.07424/) (2021)
- [A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02430/) (2023)
- [Off-policy Distributional Q(λ): Distributional RL without Importance Sampling](https://mlsanity.com/paper/2402.05766/) (2024)
- [Accelerated Distributional Temporal Difference Learning with Linear Function Approximation](https://mlsanity.com/paper/2511.12688/) (2025)
- [A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metric](https://mlsanity.com/paper/2603.12576/) (2026)
- [Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent](https://mlsanity.com/paper/2602.06567/) (2026)
- [Distributional Soft Bellman Operator under the Cram'er Geometry](https://mlsanity.com/paper/2607.17897/) (2026)

## Abstract (arXiv)

Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited. One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cram'er distance, but their results only apply to the tabular setting and ignore C51's use of a softmax to produce normalized distributions. In this paper we adapt the Cram'er distance to deal with arbitrary vectors. From it we derive a new distributional algorithm which is fully Cram'er-based and can be combined to linear function approximation, with formal guarantees in the context of policy evaluation. In allowing the model's prediction to be any real vector, we lose the probabilistic interpretation behind the method, but otherwise maintain the appealing properties of distributional approaches. To the best of our knowledge, ours is the first proof of convergence of a distributional algorithm combined with function approximation. Perhaps surprisingly, our results provide evidence that Cram'er-based distributional methods may perform worse than directly approximating the value function.