mlsanityData to 14 Oct 2026

The Geometry of Nonlinear Reinforcement Learning

arXiv 2509.01432 ↗First posted 1 Sep 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reward maximization, safe exploration, and intrinsic motivation are often studied as separate objectives in reinforcement learning (RL). We present a unified geometric framework, that views these goals as instances of a single optimization problem on the space of achievable long-term behavior in an environment. Within this framework, classical methods such as policy mirror descent, natural policy gradient, and trust-region algorithms naturally generalize to nonlinear utilities and convex constraints. We illustrate how this perspective captures robustness, safety, exploration, and diversity objectives, and outline open challenges at the interface of geometry and deep RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 57 references that are in the map. First posted on arXiv 1 Sep 2025.

Builds on

37 of 57 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  7. 2018Maximum a Posteriori Policy Optimisationcited by 328
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2016Variational Intrinsic Controlcited by 267
  10. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  11. 2017A unified view of entropy-regularized Markov decision processescited by 182
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
25 more
  1. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  2. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  3. 2021Learning One Representation to Optimize All Rewardscited by 115
  4. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  5. 2020Mirror Descent Policy Optimizationcited by 106
  6. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  7. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  8. 2021Reward is enough for convex MDPscited by 84
  9. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
  10. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  11. 2019Reinforcement Learning with Convex Constraintscited by 60
  12. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  13. 2013The Information Geometry of Mirror Descentcited by 54
  14. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  15. 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
  16. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  17. 2018Hessian Riemannian gradient flows in convex programmingcited by 26
  18. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  19. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Criticscited by 13
  20. 2022Geometry and convergence of natural policy gradient methodscited by 12
  21. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 10
  22. 2024The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processescited by 3
  23. 2024Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learningcited by 3
  24. 2025Central Path Proximal Policy Optimizationcited by 3
  25. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Active Inference as a Convex Markov Decision Processcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning
  2. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
  3. 2026Mathematical methods of reinforcement learning
  4. 2024Geometric Fabrics: a Safe Guiding Medium for Policy Learning
  5. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
  6. 2025Reward Models in Deep Reinforcement Learning: A Survey
  7. 2026Hereditary Geometric Meta-RL: Nonlocal Generalization via Task Symmetries
  8. 2024Combining Automated Optimisation of Hyperparameters and Reward Shape