# Physical policy gradient theorem for in situ stochastic-adjoint training

arXiv: https://arxiv.org/abs/2609.05808 · HTML: https://mlsanity.com/paper/2609.05808/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Training of photonic neural networks through in situ backpropagation](https://mlsanity.com/paper/1805.09943/) (2018)
- [Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks](https://mlsanity.com/paper/2205.08501/) (2022)
- [Nonlinear optical encoding enabled by recurrent linear scattering](https://mlsanity.com/paper/2307.08558/) (2023)
- [In-situ Physical Adjoint Computing in multiple-scattering electromagnetic environments for wave control](https://mlsanity.com/paper/2503.21107/) (2025)
- [Adaptive Sensing beyond Non-Adaptive Information Limits: End-to-End Co-Design of Geometry, Policy, and Inference](https://mlsanity.com/paper/2604.25193/) (2026)
- [In-situ Adjoint Wave Control in Reconfigurable Non-Hermitian Nonlinear Systems](https://mlsanity.com/paper/2608.13503/) (2026)

## Abstract (arXiv)

In situ adjoint training extracts parameter gradients directly from measurement, but has so far been limited to reciprocal or restricted systems. Here, we introduce the physical counterpart of the policy gradient theorem: a stochastic-adjoint gradient estimator that lifts these constraints by trading reciprocity for nondegenerate diffusion. As validation, we train a nonlinear resonator network, whose own dynamics supply the policy, against antagonistic temporal modulations with gradients from measured stochastic trajectories alone, without finite differences or a separate adjoint experiment.