mlsanityData to 7 Oct 2026

OPERA: Operator-residual feedback for reliable autonomous optical experiments with language-model agents

arXiv 2608.05990 ↗First posted 6 Aug 2026cs.AI cs.CE physics.app-phcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Autonomous agents choose actions using scores that may not reflect experimental success. We developed OPERA, an operator-residual framework for optical experiments. It represents experimental actions as optical operators and evaluates their outcomes using physically interpretable residuals. Operators specify executable changes to measurement, control or reconstruction, while residuals report departures from specified physical conditions. The agent uses both to select, combine or generate operators, and physical performance is evaluated independently against a withheld reference. Across three optical tasks, score-only feedback produced score increases without physical improvement in 23.6--39.0% of decisions, compared with 0.9--1.9% for operator-residual feedback. Operator-residual feedback increased the probability of reaching and maintaining task targets and reduced experimental budgets. Protocols selected in digital twins were transferred to three optical instruments, and repeated experiments showed a lower projection budget in structured-light reconstruction. Together, operators and residuals guide autonomous decisions using measurable physical evidence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 39 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

16 of 39 references.

References · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  8. 2017Inverse Reward Designcited by 246
  9. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  10. 2018Categorizing Variants of Goodhart's Lawcited by 132
  11. 2023Modern Bayesian Experimental Designcited by 121
  12. 2019Variational Bayesian Optimal Experimental Designcited by 116
4 more
  1. 2017Cautious Model Predictive Control using Gaussian Process Regressioncited by 91
  2. 2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 91
  3. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  4. 2025Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materialscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Agents for Active Perception in Optics Laboratories
  2. 2026OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design
  3. 2025NEBULA: Do We Evaluate Vision-Language-Action Agents Correctly?
  4. 2026The Observability Gap: Why Output-Level Human Feedback Fails for LLM Coding Agents
  5. 2026MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
  6. 2026RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
  7. 2025Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization
  8. 2026Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks