mlsanityData to 14 Oct 2026

Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces

arXiv 2508.14102 ↗First posted 16 Aug 2025cs.LG cs.RO cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Trust region-based optimization methods have become foundational reinforcement learning algorithms that offer stability and strong empirical performance in continuous control tasks. Growing interest in scalable and reusable control policies translate also in a demand for morphological generalization, the ability of control policies to cope with different kinematic structures. Graph-based policy architectures provide a natural and effective mechanism to encode such structural differences. However, while these architectures accommodate variable morphologies, the behavior of trust region methods under varying action space dimensionality remains poorly understood. To this end, we conduct a theoretical analysis of trust region-based policy optimization methods, focusing on both Trust Region Policy Optimization (TRPO) and its widely used first-order approximation, Proximal Policy Optimization (PPO). The goal is to demonstrate how varying action space dimensionality influence the optimization landscape, particularly under the constraints imposed by KL-divergence or policy clipping penalties. Complementing the theoretical insights, an empirical evaluation under morphological variation is carried out using the Gymnasium Swimmer environment. This benchmark offers a systematically controlled setting for varying the kinematic structure without altering the underlying task, making it particularly well-suited to study morphological generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 33 references that are in the map. First posted on arXiv 16 Aug 2025.

Builds on

27 of 33 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  7. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  8. 2018Graph networks as learnable physics engines for inference and controlcited by 244
  9. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  10. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  11. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  12. 2020Rapidly Adaptable Legged Robots via Evolutionary Meta-Learningcited by 43
15 more
  1. 2018Policy Optimization as Wasserstein Gradient Flowscited by 34
  2. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  3. 2019Graph Policy Gradients for Large Scale Robot Controlcited by 28
  4. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 25
  5. 2024Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Maskingcited by 19
  6. 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
  7. 2018An Adaptive Clipping Approach for Proximal Policy Optimizationcited by 14
  8. 2022Meta Reinforcement Learning for Optimal Design of Legged Robotscited by 13
  9. 2025Wasserstein Policy Optimizationcited by 13
  10. 2019Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learningcited by 12
  11. 2022Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actionscited by 8
  12. 2023Graph Reinforcement Learning for Network Control via Bi-Level Optimizationcited by 8
  13. 2023Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policiescited by 8
  14. 2021Decaying Clipping Range in Proximal Policy Optimizationcited by 6
  15. 2024HeteroMorpheus: Universal Control Based on Morphological Heterogeneity Modelingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
  2. 2023Supported Trust Region Optimization for Offline Reinforcement Learning
  3. 2024Embedding Safety into RL: A New Take on Trust Region Methods
  4. 2026Ratio-Variance Regularized Policy Optimization
  5. 2025PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization
  6. 2025Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learning
  7. 2026Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization
  8. 2024Geometric Fabrics: a Safe Guiding Medium for Policy Learning