mlsanityData to 14 Oct 2026

The Value Function Polytope in Reinforcement Learning

arXiv 1901.11524 ↗First posted 31 Jan 2019cs.LG cs.AI stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structural relationship between policies and value functions including the line theorem, which shows that the value functions of policies constrained on all but one state describe a line segment. Finally, we use this novel perspective to introduce visualizations to enhance the understanding of the dynamics of reinforcement learning algorithms.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 0; 2024: 6; 2025: 6; 2026: 4.

29 papers.

Sort:

By area:

  1. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  2. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
  3. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  4. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  5. 2021Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiationcited by 35
  6. 2021Metrics and continuity in reinforcement learningcited by 26
  7. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  8. 2022Generalized Data Distribution Iterationcited by 12
  9. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  10. 2021The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPscited by 9
  11. 2022Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learningcited by 8
  12. 2021An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learningcited by 7
17 more
  1. 2021GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learningcited by 7
  2. 2022The Geometry of Robust Value Functionscited by 6
  3. 2022Geometric Policy Iteration for Markov Decision Processescited by 3
  4. 2024MDP Geometry, Normalization and Reward Balancing Solverscited by 3
  5. 2024Functional Acceleration for Policy Mirror Descentcited by 3
  6. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  7. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  8. 2024Topological Foundations of Reinforcement Learningcited by 1
  9. 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimationcited by 1
  10. 2025Bellman operator convergence enhancements in reinforcement learning algorithmscited by 1
  11. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookaheadcited by 1
  12. 2026Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iterationcited by 1
  13. 2025Evaluation-Aware Reinforcement Learningcited by 0
  14. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  15. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0
  16. 2026The Value Function Semi-Algebraic Set in Partially Observable Markov Decision Processescited by 0
  17. 2026Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPscited by 0