mlsanityData to 14 Oct 2026

Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

arXiv 2507.20853 ↗First posted 28 Jul 2025cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Advances in reinforcement learning (RL) have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose building a theoretical understanding of continuous state and action spaces by employing a geometric lens to understand the locally attained set of states. The set of all parametrised policies learnt through a semi-gradient based approach induces a set of attainable states in RL. We show that the training dynamics of a two-layer neural policy induce a low dimensional manifold of attainable states embedded in the high-dimensional nominal state space trained using an actor-critic algorithm. We prove that, under certain conditions, the dimensionality of this manifold is of the order of the dimensionality of the action space. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments and also demonstrate the results in a toy environment with varying dimensionality. We also show the applicability of this theoretical result by introducing a local manifold learning layer to the policy and value function networks to improve the performance in control environments with very high degrees of freedom by changing one layer of the neural network to learn sparse representations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 62 of its 132 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

62 of 132 references.

References · 62

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  11. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  12. 2017Deep Neural Networks as Gaussian Processescited by 622
38 more
  1. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  9. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  10. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  11. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  12. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  13. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 142
  14. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  15. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  16. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  17. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  18. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  19. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  20. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  21. 2022q-Learning in Continuous Timecited by 91
  22. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  23. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  24. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  25. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  26. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  27. 2019Deep ReLU network approximation of functions on a manifoldcited by 74
  28. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  29. 2015Provable approximation properties for deep neural networkscited by 60
  30. 2019Convergence of Adversarial Training in Overparametrized Neural Networkscited by 58
  31. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 50
  32. 2017Robust Locally-Linear Controllable Embeddingcited by 38
  33. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  34. 2019Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitivescited by 30
  35. 2020Geometric compression of invariant manifolds in neural netscited by 28
  36. 2020Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Controlcited by 28
  37. 2021Metrics and continuity in reinforcement learningcited by 26
  38. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 24

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  2. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1
  3. 2025ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors
  2. 2026Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry
  3. 2026SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
  4. 2024Geometric Fabrics: a Safe Guiding Medium for Policy Learning
  5. 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
  6. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
  7. 2024Reinforcement Learning in Hyperbolic Spaces: Models and Experiments
  8. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space