# VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities

arXiv: https://arxiv.org/abs/2412.18161 · HTML: https://mlsanity.com/paper/2412.18161/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 85 references that are in the map.
- First posted on arXiv 24 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis](https://mlsanity.com/paper/2306.11296/) (2023)
- [AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews](https://mlsanity.com/paper/2408.10365/) (2024)
- [LLMatDesign: Autonomous Materials Discovery with Large Language Models](https://mlsanity.com/paper/2406.13163/) (2024)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation](https://mlsanity.com/paper/2407.08940/) (2024)
- [LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation](https://mlsanity.com/paper/2401.17244/) (2024)
- [Opportunities for Retrieval and Tool Augmented Large Language Models in Scientific Facilities](https://mlsanity.com/paper/2312.01291/) (2023)
- [Accelerating Scientific Discovery with Generative Knowledge Extraction, Graph-Based Representation, and Multimodal Intelligent Graph Reasoning](https://mlsanity.com/paper/2403.11996/) (2024)
- [Specifications: The missing link to making the development of LLM systems an engineering discipline](https://mlsanity.com/paper/2412.05299/) (2024)
- [Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Design](https://mlsanity.com/paper/2401.13803/) (2024)
- [Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper](https://mlsanity.com/paper/2408.10680/) (2024)
- [From Intention To Implementation: Automating Biomedical Research via LLMs](https://mlsanity.com/paper/2412.09429/) (2024)
- [Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2410.13768/) (2024)
- [Large Language Model-Guided Prediction Toward Quantum Materials Synthesis](https://mlsanity.com/paper/2410.20976/) (2024)

## Built on by (most cited first, 10 of 10)

- [Agentic Artificial Intelligence for Multistage Physics Experiments at a Large-Scale User Facility Particle Accelerator](https://mlsanity.com/paper/2509.17255/) (2025)
- [Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition](https://mlsanity.com/paper/2510.15280/) (2025)
- [Osprey: Production-Ready Agentic AI for Safety-Critical Control Systems](https://mlsanity.com/paper/2508.15066/) (2025)
- [LAP: An Agent-to-Instrument Protocol for Autonomous Science](https://mlsanity.com/paper/2606.03755/) (2026)
- [Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensions](https://mlsanity.com/paper/2509.22814/) (2025)
- [EAA: Automating materials characterization with vision language model agents](https://mlsanity.com/paper/2602.15294/) (2026)
- [VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing](https://mlsanity.com/paper/2604.19412/) (2026)
- [LLM Jaggedness Unlocks Scientific Creativity](https://mlsanity.com/paper/2605.10574/) (2026)
- [Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen tasks](https://mlsanity.com/paper/2608.05266/) (2026)
- [APEXA: Execution-Integrity Enforcement for Multi-Agent LLM Automation of Synchrotron Data Reduction](https://mlsanity.com/paper/2609.24165/) (2026)

## Abstract (arXiv)

Scientific user facilities, such as synchrotron beamlines, are equipped with a wide array of hardware and software tools that require a codebase for human-computer-interaction. This often necessitates developers to be involved to establish connection between users/researchers and the complex instrumentation. The advent of generative AI presents an opportunity to bridge this knowledge gap, enabling seamless communication and efficient experimental workflows. Here we present a modular architecture for the Virtual Scientific Companion (VISION) by assembling multiple AI-enabled cognitive blocks that each scaffolds large language models (LLMs) for a specialized task. With VISION, we performed LLM-based operation on the beamline workstation with low latency and demonstrated the first voice-controlled experiment at an X-ray scattering beamline. The modular and scalable architecture allows for easy adaptation to new instrument and capabilities. Development on natural language-based scientific experimentation is a building block for an impending future where a science exocortex -- a synthetic extension to the cognition of scientists -- may radically transform scientific practice and discovery.