mlsanityData to 14 Oct 2026

ChemDFM-X: Towards Large Multimodal Model for Chemistry

arXiv 2409.13194 ↗First posted 20 Sep 2024cs.LG cs.CL cs.MMcited by 16

Not on the map. Why?

Abstract (arXiv)

Rapid developments of AI tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

31 of 47 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  5. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  8. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  9. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  10. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  11. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  12. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
19 more
  1. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  2. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  3. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  4. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  5. 2024ChemLLM: A Chemical Large Language Modelcited by 106
  6. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 90
  7. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  8. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  9. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  10. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 66
  11. 2022MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generationcited by 49
  12. 2022Energy-Motivated Equivariant Pretraining for 3D Molecular Graphscited by 37
  13. 2022A Systematic Survey of Chemical Pre-trained Modelscited by 34
  14. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 32
  15. 2021Dual-view Molecule Pre-trainingcited by 27
  16. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  17. 2024MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognitioncited by 16
  18. 2023ReactionT5: a large-scale pre-trained model towards application of limited reaction datacited by 13
  19. 2024Multi-Modal Representation Learning for Molecular Property Prediction: Sequence, Graph, Geometrycited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  2. 2025ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledgecited by 20
  3. 2025ChemMLLM: Chemical Multimodal Large Language Modelcited by 17
  4. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  5. 2025TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Taskscited by 6
  6. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 6
  7. 2025RetroDFM-R: Reasoning-Driven Retrosynthesis Prediction with Large Language Models via Reinforcement Learningcited by 3
  8. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  9. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  10. 2025RTMol: Rethinking Molecule-text Alignment in a Round-trip Viewcited by 1
  11. 2026MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognitioncited by 1
  12. 2026MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognitioncited by 1
4 more
  1. 2025Evaluating Hydro-Science and Engineering Knowledge of Large Language Modelscited by 0
  2. 2025Accelerating Materials Discovery: Learning a Universal Representation of Chemical Processes for Cross-Domain Property Predictioncited by 0
  3. 2026RxnNano:Training Compact LLMs for Chemical Reaction and Retrosynthesis Prediction via Hierarchical Curriculum Learningcited by 0
  4. 2026Localize, Then Reason: Visual Latent Structural Reasoning for Molecular Properties and Editscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
  2. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
  3. 2025Assessing the Chemical Intelligence of Large Language Models
  4. 2025ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
  5. 2024Probing the limitations of multimodal language models for chemistry and materials research
  6. 2024Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation
  7. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
  8. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistry