# Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Design

arXiv: https://arxiv.org/abs/2401.13803 · HTML: https://mlsanity.com/paper/2401.13803/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2024.

## Built on by (most cited first, 9 of 9)

- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)
- [34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery](https://mlsanity.com/paper/2505.03049/) (2025)
- [Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials](https://mlsanity.com/paper/2504.10281/) (2025)
- [Towards a Science Exocortex](https://mlsanity.com/paper/2406.17809/) (2024)
- [Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy](https://mlsanity.com/paper/2506.08423/) (2025)
- [Human-AI Collaborative Autonomous Experimentation With Proxy Modeling for Comparative Observation](https://mlsanity.com/paper/2603.12618/) (2026)
- [Agentic AI for operating scientific instruments for nanoscale characterization](https://mlsanity.com/paper/2608.26198/) (2026)
- [Human-agent discovery of reconfigurable in-plane ferroelectric superdomain control](https://mlsanity.com/paper/2609.06887/) (2026)

## Abstract (arXiv)

With the advent of large language models (LLMs), in both the open source and proprietary domains, attention is turning to how to exploit such artificial intelligence (AI) systems in assisting complex scientific tasks, such as material synthesis, characterization, analysis and discovery. Here, we explore the utility of LLM, particularly ChatGPT4, in combination with application program interfaces (APIs) in tasks of experimental design, programming workflows, and data analysis in scanning probe microscopy, using both in-house developed API and API given by a commercial vendor for instrument control. We find that the LLM can be especially useful in converting ideations of experimental workflows to executable code on microscope APIs. Beyond code generation, we find that the GPT4 is capable of analyzing microscopy images in a generic sense. At the same time, we find that GPT4 suffers from inability to extend beyond basic analyses or more in-depth technical experimental design. We argue that a LLM specifically fine-tuned for individual scientific domains can potentially be a better language interface for converting scientific ideations from human experts to executable workflows, such a synergy between human expertise and LLM efficiency in experimentation can open new door for accelerating scientific research, enabling effective experimental protocols archive and sharing in scientific community.