# Vision-Language Agents for Active Perception in Optics Laboratories

arXiv: https://arxiv.org/abs/2609.32918 · HTML: https://mlsanity.com/paper/2609.32918/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 6 of 6)

- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language](https://mlsanity.com/paper/2405.08888/) (2024)
- [AI-Driven Robotics for Optics](https://mlsanity.com/paper/2505.17985/) (2025)
- [A Framework for Closed-Loop Robotic Assembly, Alignment and Self-Recovery of Precision Optical Systems](https://mlsanity.com/paper/2603.21496/) (2026)
- [CLUSTER: Derivative-free optimization of smooth functions with parameter-change costs](https://mlsanity.com/paper/2606.20498/) (2026)
- [A Programmable Optics Cloud Laboratory](https://mlsanity.com/paper/2609.16413/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.