# OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

arXiv: https://arxiv.org/abs/2602.23761 · HTML: https://mlsanity.com/paper/2602.23761/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)
- [LLMLight: Large Language Models as Traffic Signal Control Agents](https://mlsanity.com/paper/2312.16044/) (2023)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Exploring Quasi-Global Solutions to Compound Lens Based Computational Imaging Systems](https://mlsanity.com/paper/2404.19201/) (2024)
- [Neuro-inspired automated lens design](https://mlsanity.com/paper/2510.09979/) (2025)
- [Design, analysis, and manufacturing of a glass-plastic hybrid minimalist aspheric panoramic annular lens](https://mlsanity.com/paper/2405.02942/) (2024)

## Built on by (most cited first, 3 of 3)

- [Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks](https://mlsanity.com/paper/2607.07189/) (2026)
- [HALO: A Physics-Aware LLM Agent Framework for Nanophotonic Design](https://mlsanity.com/paper/2608.28877/) (2026)
- [NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents](https://mlsanity.com/paper/2610.03631/) (2026)

## Abstract (arXiv)

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.