mlsanityData to 7 Oct 2026

GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

arXiv 2308.06911 ↗First posted 14 Aug 2023cs.LG cs.CL q-bio.BMcited by 82

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 63 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

39 of 63 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
27 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  10. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  11. 2022A Generalist Agentcited by 802
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  17. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  18. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  19. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  20. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  21. 2022Translation between Molecules and Natural Languagecited by 173
  22. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  23. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  24. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  25. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  26. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 91
  27. 2022Foundation Transformerscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 42; 2025: 22; 2026: 12.

82 papers.

Sort:

By area:

  1. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  2. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  3. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  4. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  5. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  6. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 84
  7. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  8. 2024A Survey of Large Language Models for Graphscited by 71
  9. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  10. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  11. 2024BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuningcited by 54
  12. 2024Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challengescited by 41
38 more
  1. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  2. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  3. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  4. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  5. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  6. 2024LLaMo: Large Language Model-based Molecular Graph Assistantcited by 22
  7. 2024L+M-24: Building a Dataset for Language + Molecules @ ACL 2024cited by 21
  8. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  9. 2024A quantitative analysis of knowledge-learning preferences in large language models in molecular sciencecited by 18
  10. 2024HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignmentcited by 17
  11. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 16
  12. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 16
  13. 2024PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomescited by 14
  14. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  15. 2026Graph is a Substrate Across Data Modalitiescited by 13
  16. 2024GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaborationcited by 12
  17. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12
  18. 2024LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Modelscited by 11
  19. 2024Learning Multi-view Molecular Representations with Structured and Unstructured Knowledgecited by 11
  20. 2024Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustnesscited by 11
  21. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  22. 2025HyperG: Hypergraph-Enhanced LLMs for Structured Knowledgecited by 11
  23. 20243D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modelingcited by 9
  24. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  25. 2025OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discoverycited by 8
  26. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  27. 2024Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Surveycited by 6
  28. 2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilizationcited by 6
  29. 2025KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledgecited by 6
  30. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 5
  31. 2024Bridging Text and Molecule: A Survey on Multimodal Frameworks for Moleculecited by 4
  32. 2025Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Spacecited by 4
  33. 2024Instruction-Based Molecular Graph Generation with Unified Text-Graph Diffusion Modelcited by 3
  34. 2024The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimationcited by 3
  35. 2024Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrievalcited by 3
  36. 2024Property Enhanced Instruction Tuning for Multi-task Molecule Generation with Large Language Modelscited by 3
  37. 2025LLM-Fusion: A Novel Multimodal Fusion Model for Accelerated Material Discoverycited by 3
  38. 2025MolEdit: Knowledge Editing for Multimodal Molecule Language Modelscited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Molecular Representations for Large Language Models
  2. 2025BioMedGPT-Mol: Multi-task Learning for Molecular Understanding and Generation
  3. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models
  4. 2025ChemMLLM: Chemical Multimodal Large Language Model
  5. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
  6. 2024MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts
  7. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Study
  8. 2025Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens