OmniScientist: An Omni-Modal Omni-Discipline AI Scientist
Not on the map. Why?
Abstract (arXiv)
Recent advances in foundation models have enabled AI scientists to automate increasingly complete research workflows, from hypothesis generation and code execution to manuscript preparation. Yet workflow coverage alone does not provide access to the full evidence on which scientific discovery depends. Existing systems typically reason over text, code, labels, or precomputed summaries, leaving scientifically decisive spatial, temporal, cross-channel, and procedural relations unavailable to the agent. We introduce OmniScientist, an end-to-end, omni-modal AI scientist that conducts multidisciplinary research directly from heterogeneous raw evidence. A perception layer and 3 autonomous agents for ideation, experiment, and writeup operate within a deterministic pipeline, allowing observations to shape research questions, experimental decisions, and final claims throughout the research lifecycle. By running idea, rigour, and claim checks in code, the system enforces novelty screening, statistical validity, execution provenance, and numerical traceability. We evaluate OmniScientist on 36 real-data cases spanning 5 discipline families, 4 families of scientific evidence, and modalities including images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs. The system completes the full path from raw data to a compiled manuscript in all 36 cases and achieves a mean overall paper score of 6.3 with the reference reasoning backbone. In paired comparisons against a blind variant that receives only precomputed scalar features, direct perception improves all 7 evaluation dimensions and wins 85% of head-to-head judgments. These results show that lifecycle-wide perception is essential for evidence-grounded scientific discovery and provides a practical path toward broadly capable AI scientists.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 57 of its 95 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
57 of 95 references.
References · 57
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Visual Instruction Tuningcited by 7,792
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
38 more
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
- 2019AI Feynman: a Physics-Inspired Method for Symbolic Regressioncited by 536
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
- 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
- 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
- 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 375
- 2025Accelerating scientific discovery with Co-Scientistcited by 353
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
- 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
- 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
- 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
- 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 166
- 2026Towards End-to-End Automation of AI Researchcited by 150
- 2015An open access repository of images on plant health to enable the development of mobile disease diagnosticscited by 132
- 2025AI-Researcher: Autonomous Scientific Innovationcited by 88
- 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 78
- 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
- 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
- 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 60
- 2019MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspectioncited by 57
- 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
- 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
- 2018Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challengecited by 44
- 2025InternAgent: When Agent Becomes the Scientist -- Building Closed-Loop System from Hypothesis to Verificationcited by 41
- 2021Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxiescited by 40
- 2018A Commute in Data: The comma2k19 Datasetcited by 33
- 2024NatureLM-audio: an Audio-Language Foundation Model for Bioacousticscited by 32
- 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
- 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
- 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
- 2022The Caltech Fish Counting Dataset: A Benchmark for Multiple-Object Tracking and Countingcited by 24
- 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
- 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 23
- 2025The Denario project: Deep knowledge AI agents for scientific discoverycited by 21
- 2015WHOI-Plankton- A Large Scale Fine Grained Visual Recognition Benchmark Dataset for Plankton Classificationcited by 17
- 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understandingcited by 15
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.