mlsanityData to 14 Oct 2026

Monkey King Bang: A Unified Scientific Multimodal Foundation Model

arXiv 2607.20557 ↗First posted 17 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Scientific discovery is increasingly shifting from isolated disciplines to multi-domain reasoning, and AI for science faces a similar transition. Existing systems are either specialised for individual domains or unify scientific data mainly through text tokenisation and prompt-based interfaces, limiting their ability to handle diverse scientific inputs, produce modality-native outputs, and support joint understanding, reasoning, and generation across scientific domains. We introduce MKB, a unified scientific multimodal model for both understanding and generation, built around a shared Transformer backbone and modality-tailored encoders, adapters, and decoders. MKB covers six scientific branches, including DNA, RNA, proteins, small molecules, earth science, and medical images, and supports native outputs such as biological sequences, molecular strings, meteorological fields, and segmentation masks. Training follows a two-stage modality-then-language curriculum: Stage 1 aligns modality-specific components with the frozen backbone, and Stage 2 consolidates them with the language backbone using mixed scientific and general corpora. Experiments show that MKB achieves competitive scientific understanding across biological and molecular benchmarks, produces high-fidelity native outputs for weather forecasting, biological generation, and medical-image segmentation, and largely retains the general capabilities of its Qwen3-VL backbone. These results demonstrate the feasibility of the proposed paradigm, suggesting that shared-backbone models with modality-tailored components can provide a promising foundation for future cross-domain scientific multimodal exploration. The model and code are publicly available at https://github.com/Shanghai-Academy-of-AI-For-Science/MKB and https://huggingface.co/sais-org/MKB.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

20 of 32 references.

References · 20

  1. 2023Segment Anythingcited by 8,234
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  7. 2025SAM 3: Segment Anything with Conceptscited by 906
  8. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 776
  9. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  10. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  11. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 241
  12. 2022Translation between Molecules and Natural Languagecited by 177
8 more
  1. 2018DeepDTA: Deep Drug-Target Binding Affinity Predictioncited by 173
  2. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 143
  3. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  4. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
  5. 2025Nature Language Model: Deciphering the Language of Nature for Scientific Discoverycited by 36
  6. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 29
  7. 2024Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Modelscited by 10
  8. 2026Suiren-1.0 Technical Report: A Family of Molecular Foundation Modelscited by 1

Built on by

Not cited yet.