# Operator-Guided Invariance Learning for Continuous Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.06500 · HTML: https://mlsanity.com/paper/2605.06500/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [LieTransformer: Equivariant self-attention for Lie Groups](https://mlsanity.com/paper/2012.10885/) (2020)
- [Modeling Other Players with Bayesian Beliefs for Games with Incomplete Information](https://mlsanity.com/paper/2405.14122/) (2024)
- [Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents](https://mlsanity.com/paper/2602.02995/) (2026)
- [Network Diffuser for Placing-Scheduling Service Function Chains with Inverse Demonstration](https://mlsanity.com/paper/2501.05673/) (2025)
- [Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Tasks](https://mlsanity.com/paper/2502.00633/) (2025)
- [Geometry of Drifting MDPs with Path-Integral Stability Certificates](https://mlsanity.com/paper/2601.21991/) (2026)
- [Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2602.02900/) (2026)
- [Structuring Value Representations via Geometric Coherence in Markov Decision Processes](https://mlsanity.com/paper/2602.02978/) (2026)
- [MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning](https://mlsanity.com/paper/2511.06142/) (2025)
- [Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics](https://mlsanity.com/paper/2602.06939/) (2026)

## Built on by (most cited first, 5 of 5)

- [Metric-Gradient Projection for Stable Multi-Agent Policy Learning](https://mlsanity.com/paper/2605.18809/) (2026)
- [Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry](https://mlsanity.com/paper/2605.14304/) (2026)
- [Learning Not to Optimize: Physics-Informed Action-Space Reshaping for Intent-Based Network Control](https://mlsanity.com/paper/2608.00908/) (2026)
- [Differentiable Policy Transport over Multi-Layer Network Feasibility Geometry](https://mlsanity.com/paper/2609.26068/) (2026)
- [Geometry-Aware Operator Families for Structured Representation Learning](https://mlsanity.com/paper/2609.33089/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) with continuous time and state/action spaces is often data-intensive and brittle under nuisance variability and shift, motivating methods that exploit value-preserving structures to stabilize and improve learning. Most existing approaches focus on special cases, such as prescribed symmetries and exact equivariance, without addressing how to discover more general structures that require nonlinear operators to transform and map between continuous state/action systems with isomorphic value functions. We propose VPSD-RL (Value-Preserving Structure Discovery for Reinforcement Learning). It models continuous RL as a controlled diffusion with value-preserving mappings defined through Lie-group actions and associated pullback operators. We show that a value-preserving structure exists exactly when pulling back the value function and pushing forward actions commute with the controlled generator and reward functional. Further, approximate value-preserving structures with rigorous guarantees can be found when the Hamilton--Jacobi--Bellman mismatch is small. This framework discovers exact and approximate value-preserving structures by searching for the associated Lie group operators. VPSD-RL fits differentiable drift, diffusion, and reward models; learns infinitesimal generators via determining-equation residual minimization; exponentiates them with ODE flows to obtain finite transformations; and integrates them into continuous RL through transition augmentation and transformation-consistency regularization. We show that bounded generator/reward mismatch implies quantitative stability of the optimal value function along approximate orbits, with sensitivity governed by the effective horizon, and observe improved data efficiency and robustness on continuous-control benchmarks.