MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Multimodal large language models (MLLMs) show remarkable potential for scientific reasoning, yet their performance in specialized domains such as microscopy remains limited by the scarcity of domain-specific training data and the difficulty of encoding fine-grained expert knowledge into model parameters. To bridge the gap, we introduce MicroWorld, a framework that constructs a multimodal attributed property graph (MAPG) from large-scale scientific image--caption corpora and leverages it to augment MLLM reasoning at inference time without any domain-specific fine-tuning. MicroWorld extracts biomedical entities and relations via scispaCy or LLM-based triplet mining, aligns images and entities in a shared embedding space using Qwen3-VL-Embedding, and assembles a knowledge graph comprising approximately 111K nodes and 346K typed edges spanning eight relation categories. At inference time, a graph-augmented retrieval pipeline matches query entities to the MAPG and injects structured knowledge context into the MLLM prompt. On the MicroVQA benchmark, MicroWorld improves the reasoning performance of Qwen3-VL-8B-Instruct by 37.5%, outperforming GPT-5 by 13.0% to achieve a new state-of-the-art. Furthermore, it yields a 6.0% performance gain on the MicroBench benchmark. Extensive experiments demonstrate the enhanced generalization capability introduced by MicroWorld. A qualitative case study further reveals both the mechanisms through which structured knowledge improves reasoning and the failure modes that point to promising future directions. Code and data are available at https://github.com/ieellee/MicroWorld.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 49 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
29 of 49 references.
Closest ancestors · 2
Same direction, newest first
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
- 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research17 Mar
Other references · 27
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2025OpenAI GPT-5 System Cardcited by 989
- 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
- 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 315
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
- 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
15 more
- 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
- 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
- 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
- 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
- 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
- 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 123
- 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
- 2021Graph Representation Learning in Biomedicinecited by 80
- 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
- 2025KGGen: Extracting Knowledge Graphs from Plain Text with Language Modelscited by 49
- 2018LinkNet: Relational Embedding for Scene Graphcited by 45
- 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
- 2024Multimodal Large Language Models for Bioimage Analysiscited by 6
- 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understandingcited by 5
- 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQAcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model
- 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
- 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
- 2026S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding
- 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs
- 2024Enhancing Explainability in Multimodal Large Language Models Using Ontological Context
- 2026SmoGVLM: A Small, Graph-enhanced Vision-Language Model
- 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning