mlsanityData to 14 Oct 2026

ChemMLLM: Chemical Multimodal Large Language Model

arXiv 2505.16326 ↗First posted 22 May 2025cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recent years have seen rapid progress in multimodal large language models (MLLMs) in the field of chemistry. However, chemical MLLMs that can handle cross-modal understanding and generation remain underexplored. To fill this gap, we propose ChemMLLM, a unified chemical multimodal large language model for molecule understanding and generation. In this work, we design five types of multimodal tasks across text, molecular SMILES strings and images, and curate the datasets. We benchmark ChemMLLM against a range of general leading MLLMs, Chemical LLMs and specialized models on these tasks. Experimental results show that ChemMLLM achieves superior performance among general-purpose MLLMs and close performance to specialized models across all evaluated tasks. Our work extends the capabilities of chemical multimodal large language models to the realm of image generation, demonstrating the feasibility of unifying multiple cross-modal chemical tasks within a single foundation model and enabling more intuitive, visual human-AI interaction.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 85 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

43 of 85 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area14 Aug

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
30 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  4. 2018Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Modelscited by 275
  5. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  6. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  7. 2020Hierarchical Generation of Molecular Graphs using Structural Motifscited by 208
  8. 2022Translation between Molecules and Natural Languagecited by 177
  9. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 166
  10. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  11. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  12. 2024ChemLLM: A Chemical Large Language Modelcited by 106
  13. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  14. 2018Learning Multimodal Graph-to-Graph Translation for Molecular Optimizationcited by 88
  15. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  16. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  17. 2024BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuningcited by 54
  18. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 52
  19. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  20. 2022MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generationcited by 49
  21. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  22. 2019CORE: Automatic Molecule Optimization Using Copy & Refine Strategycited by 27
  23. 2023Automated 3D Pre-Training for Molecular Property Predictioncited by 24
  24. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  25. 2017ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity?cited by 21
  26. 2017Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Modelscited by 19
  27. 2024HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignmentcited by 17
  28. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  29. 2024Learning Multi-view Molecular Representations with Structured and Unstructured Knowledgecited by 11
  30. 2024HGTDR: Advancing Drug Repurposing with Heterogeneous Graph Transformerscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  3. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  4. 2025ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledgecited by 20
  5. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 16
  6. 2025SciReasoner: Laying the Scientific Reasoning Ground Across Disciplinescited by 11
  7. 2026LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoningcited by 8
  8. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  9. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 6
  10. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3
  11. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  12. 2025SpectrumWorld: Artificial Intelligence Foundation for Spectroscopycited by 2
5 more
  1. 2025SpecMol: A Spectroscopy-Grounded Foundation Model for Multi-Task Molecular Learningcited by 1
  2. 2026Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Languagecited by 1
  3. 2026Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervisioncited by 0
  4. 2026Localize, Then Reason: Visual Latent Structural Reasoning for Molecular Properties and Editscited by 0
  5. 2026TMCS: Tool-Grounded Multi-Agent Reasoning for Compositional Chemical Problem Solvingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
  2. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Study
  3. 2025Omni-Mol: Multitask Molecular Model for Any-to-any Modalities
  4. 2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilization
  5. 2024Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs
  6. 2024Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation
  7. 2025ChatMol: A Versatile Molecule Designer Based on the Numerically Enhanced Large Language Model
  8. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text