# When Every Simulation Counts: Value-Based Reinforcement Learning for Accelerated Photonics Inverse Design

arXiv: https://arxiv.org/abs/2607.23469 · HTML: https://mlsanity.com/paper/2607.23469/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 49 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 7 of 7)

- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Deep neural networks for the evaluation and design of photonic devices](https://mlsanity.com/paper/2007.00084/) (2020)
- [Inverse design of photonic crystals through automatic differentiation](https://mlsanity.com/paper/2003.00379/) (2020)
- [Benchmarking five global optimization approaches for nano-optical shape optimization and parameter reconstruction](https://mlsanity.com/paper/1809.06674/) (2018)
- [Inverse Design of Photonic Crystal Surface Emitting Lasers is a Sequence Modeling Problem](https://mlsanity.com/paper/2403.05149/) (2024)

## Abstract (arXiv)

Photonic-crystal surface-emitting lasers (PCSELs) can combine high-power operation with narrow-divergence surface emission, but optimizing coupled parameters requires costly full-wave simulations. Deep Q-network (DQN) optimization can reuse simulated transitions to guide edits, yet which value-learning mechanisms remain reliable under tight simulation budgets is unknown. We address this gap by comparing baseline DQN and six value-based variants for a seven-variable PCSEL design under a shared objective, simulator, 83-call budget, and four matched initializations. Beyond endpoints, we analyze sample efficiency, policy behavior, and physical response to separate learning gains from favorable starts or exploratory jumps. Dueling DQN is the only variant to improve all four seeds. Relative to the first evaluated designs, its selected structures increase the mean quality factor () from to (), reduce wavelength error by 64%, and increase upward power by 47%; compared with baseline DQN, they achieve a higher mean under the same budget. Other variants yield no consistent improvement; Double DQN reproduces baseline trajectories, while Rainbow-lite shows high upside but strong seed dependence. These results identify Dueling DQN as the most reliable configuration tested for simulation-budget-limited PCSEL inverse design and provide a reproducible framework for attributing algorithmic gains in scientific optimization. The source code is publicly available at https://github.com/Longying-Wen/PCSEL-RL.