mlsanityData to 14 Oct 2026

Environment Parameter Gradient Theorem for Co-Design in Reinforcement Learning

arXiv 2607.12590 ↗First posted 14 Jul 2026eess.SY cs.LG cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning (RL) is traditionally concerned with learning a control policy for a fixed environment. In many engineering systems, however, the environment itself is alterable, i.e., physical or operational parameters can be tuned to shape the system's transition dynamics and costs experienced by the RL agent. This motivates jointly optimizing both the policy and the environment design parameters. To this end, we establish an Environment Parameter Gradient Theorem --- a formal expression for the gradient of the RL's objective function with respect to environment parameters. The key theoretical device is a generalized action-value function Qπ,xi(s,a,zeta), which comprises two copies of the environment parameters: zeta governs the cost and transition dynamics at the current state--action pair, while xi governs the future rollouts. This decoupling yields a tractable closed-form gradient expression and is essential to the theorem's derivation. Building on this result, we develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters. We demonstrate the efficacy of our framework on a UAV network design problem, where the optimal UAV placement (environment parameters) and communication routes (governed by the policy) are learned jointly to minimize the total communication cost in the network.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 21 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

11 of 21 references.

References · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  3. 2018Reinforcement Learning for Improving Agent Designcited by 56
  4. 2018Jointly Learning to Construct and Control Agents using Deep Reinforcement Learningcited by 55
  5. 2022Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robotscited by 47
  6. 2019Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learningcited by 37
  7. 2023MORPH: Design Co-optimization with Reinforcement Learning via a Differentiable Hardware Model Proxycited by 8
  8. 2020Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Frameworkcited by 6
  9. 2024Reinforcement Learning for Efficient Design and Control Co-optimisation of Energy Systemscited by 3
  10. 2026Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimizationcited by 3
  11. 2020Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design
  2. 2026Parameter Exploration for RLVR via Variational Learning
  3. 2025The Role of Environment Access in Agnostic Reinforcement Learning
  4. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  5. 2026PACE: Parameter Change for Unsupervised Environment Design
  6. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
  7. 2026Composable Model-Free RL for Navigation with Input-Affine Systems
  8. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies