mlsanityData to 7 Oct 2026

VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities

arXiv 2412.18161 ↗First posted 24 Dec 2024cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Scientific user facilities, such as synchrotron beamlines, are equipped with a wide array of hardware and software tools that require a codebase for human-computer-interaction. This often necessitates developers to be involved to establish connection between users/researchers and the complex instrumentation. The advent of generative AI presents an opportunity to bridge this knowledge gap, enabling seamless communication and efficient experimental workflows. Here we present a modular architecture for the Virtual Scientific Companion (VISION) by assembling multiple AI-enabled cognitive blocks that each scaffolds large language models (LLMs) for a specialized task. With VISION, we performed LLM-based operation on the beamline workstation with low latency and demonstrated the first voice-controlled experiment at an X-ray scattering beamline. The modular and scalable architecture allows for easy adaptation to new instrument and capabilities. Development on natural language-based scientific experimentation is a building block for an impending future where a science exocortex -- a synthetic extension to the cognition of scientists -- may radically transform scientific practice and discovery.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 85 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

33 of 85 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  12. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
21 more
  1. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  2. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  3. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 61
  4. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 60
  5. 2023ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesiscited by 50
  6. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 42
  7. 2024LLMatDesign: Autonomous Materials Discovery with Large Language Modelscited by 41
  8. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  9. 2024Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluationcited by 33
  10. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillationcited by 32
  11. 2023Opportunities for Retrieval and Tool Augmented Large Language Models in Scientific Facilitiescited by 27
  12. 2024Accelerating Scientific Discovery with Generative Knowledge Extraction, Graph-Based Representation, and Multimodal Intelligent Graph Reasoningcited by 27
  13. 2024Specifications: The missing link to making the development of LLM systems an engineering disciplinecited by 17
  14. 2024Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Designcited by 9
  15. 2024Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whispercited by 9
  16. 2024From Intention To Implementation: Automating Biomedical Research via LLMscited by 7
  17. 2024Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systemscited by 6
  18. 2024Large Language Model-Guided Prediction Toward Quantum Materials Synthesiscited by 5
  19. 2023Domain-specific ChatBots for Science using Embeddingscited by 4
  20. 2024Towards a Science Exocortexcited by 3
  21. 2023Virtual Scientific Companion for Synchrotron Beamlines: A Prototypecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Agentic Artificial Intelligence for Multistage Physics Experiments at a Large-Scale User Facility Particle Acceleratorcited by 6
  2. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  3. 2025Osprey: Production-Ready Agentic AI for Safety-Critical Control Systemscited by 4
  4. 2026LAP: An Agent-to-Instrument Protocol for Autonomous Sciencecited by 2
  5. 2025Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensionscited by 1
  6. 2026EAA: Automating materials characterization with vision language model agentscited by 1
  7. 2026VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editingcited by 0
  8. 2026LLM Jaggedness Unlocks Scientific Creativitycited by 0
  9. 2026Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen taskscited by 0
  10. 2026APEXA: Execution-Integrity Enforcement for Multi-Agent LLM Automation of Synchrotron Data Reductioncited by 0