# Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy

arXiv: https://arxiv.org/abs/2310.05018 · HTML: https://mlsanity.com/paper/2310.05018/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 5 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Built on by (most cited first, 11 of 11)

- [Integration of Scanning Probe Microscope with High-Performance Computing: fixed-policy and reward-driven workflows implementation](https://mlsanity.com/paper/2405.12300/) (2024)
- [SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions](https://mlsanity.com/paper/2409.12295/) (2024)
- [Unraveling the Impact of Initial Choices and In-Loop Interventions on Learning Dynamics in Autonomous Scanning Probe Microscopy](https://mlsanity.com/paper/2402.00071/) (2024)
- [Co-orchestration of Multiple Instruments to Uncover Structure-Property Relationships in Combinatorial Libraries](https://mlsanity.com/paper/2402.02198/) (2024)
- [Towards accelerating physical discovery via non-interactive and interactive multi-fidelity Bayesian Optimization: Current challenges and future opportunities](https://mlsanity.com/paper/2402.13402/) (2024)
- [Benchmarking Autonomy in Scientific Experiments: A Hierarchical Taxonomy for Autonomous Large-Scale Facilities](https://mlsanity.com/paper/2601.06978/) (2026)
- [Human-AI Collaborative Autonomous Experimentation With Proxy Modeling for Comparative Observation](https://mlsanity.com/paper/2603.12618/) (2026)
- [A Large-Language-Model Assisted Automated Scale Bar Detection and Extraction Framework for Scanning Electron Microscopic Images](https://mlsanity.com/paper/2510.11260/) (2025)
- [Novelty-Driven Target-Space Discovery in Automated Electron and Scanning Probe Microscopy](https://mlsanity.com/paper/2603.16715/) (2026)
- [From Closed-Loop Optimization to Open Decision Making: Coupled Digital Twins for Predictive and Autonomous Microscopy](https://mlsanity.com/paper/2607.05758/) (2026)
- [Spatial Action Review: A Visual Analytics Dashboard for Auditing Language-to-Action Hand-offs in Electron Microscopy](https://mlsanity.com/paper/2609.24470/) (2026)

## Abstract (arXiv)

Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.