mlsanityData to 7 Oct 2026

Model-free reinforcement learning with noisy actions for automated experimental control in optics

arXiv 2405.15421 ↗First posted 24 May 2024cs.LG physics.opticscited by 2

Not on the map. Why?

Abstract (arXiv)

Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training on simulations. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC), Truncated Quantile Critics (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 105 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

31 of 105 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2019Solving Rubik's Cube with a Robot Handcited by 699
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  11. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
19 more
  1. 2019Challenges of Real-World Reinforcement Learningcited by 282
  2. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  3. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  4. 2017Parameter Space Noise for Explorationcited by 158
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  6. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  7. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  8. 2023RoboHive: A Unified Framework for Robot Learningcited by 42
  9. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 21
  10. 2023Latent Exploration for Reinforcement Learningcited by 21
  11. 2020Interferobot: aligning an optical interferometer by a reinforcement learning agentcited by 12
  12. 2019DeepRMSA: A Deep Reinforcement Learning Framework for Routing, Modulation and Spectrum Assignment in Elastic Optical Networkscited by 7
  13. 2020Parameterized Reinforcement Learning for Optical System Optimizationcited by 6
  14. 2020Deep reinforcement learning for optical systems: A case study of mode-locked laserscited by 5
  15. 2021Self-optimizing adaptive optics control with Reinforcement Learning for high-contrast imagingcited by 4
  16. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  17. 2020Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environmentscited by 3
  18. 2023Laboratory Experiments of Model-based Reinforcement Learning for Adaptive Optics Controlcited by 2
  19. 2020Experimental phase control of a 100 laser beam array with quasi-reinforcement learning of a neural network in an error reduction loopcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026A Framework for Closed-Loop Robotic Assembly, Alignment and Self-Recovery of Precision Optical Systemscited by 1
  2. 2025Automating Experimental Optics with Sample Efficient Machine Learning Methodscited by 0