mlsanityData to 7 Oct 2026

Vision Language Model is NOT All You Need: Augmentation Strategies for Molecule Language Models

arXiv 2407.09043 ↗First posted 12 Jul 2024cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recently, there has been a growing interest among researchers in understanding molecules and their textual descriptions through molecule language models (MoLM). However, despite some early promising developments, the advancement of MoLM still trails significantly behind that of vision language models (VLM). This is because unique challenges exist apart from VLM in the field of MoLM due to 1) a limited amount of molecule-text paired data and 2) missing expertise that occurred due to the specialized areas of focus among the experts. To this end, we propose AMOLE, which 1) augments molecule-text pairs with structural similarity preserving loss, and 2) transfers the expertise between the molecules. Specifically, AMOLE enriches molecule-text pairs by sharing descriptions among structurally similar molecules with a novel structural similarity preserving loss. Moreover, we propose an expertise reconstruction loss to transfer knowledge from molecules that have extensive expertise to those with less expertise. Extensive experiments on various downstream tasks demonstrate the superiority of AMOLE in comprehending molecules and their descriptions, highlighting its potential for application in real-world drug discovery. The source code for AMOLE is available at https://github.com/Namkyeong/AMOLE.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 51 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

19 of 51 references.

References · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018How Powerful are Graph Neural Networks?cited by 4,355
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2020Understanding the Behaviour of Contrastive Losscited by 397
  7. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  8. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  11. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  12. 2023Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systemscited by 118
7 more
  1. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  2. 2021Augmentation-Free Self-Supervised Learning on Graphscited by 94
  3. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  4. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  5. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  6. 2023Conditional Graph Information Bottleneck for Molecular Relational Learningcited by 24
  7. 2023Shift-Robust Molecular Relational Learning with Causal Substructurecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  2. 2025IR-Agent: Expert-Inspired LLM Agents for Structure Elucidation from Infrared Spectracited by 4
  3. 2026Improving Large Molecular Language Model via Relation-aware Multimodal Collaborationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model
  2. 2026MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding
  3. 2025Omni-Mol: Multitask Molecular Model for Any-to-any Modalities
  4. 2024MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts
  5. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Study
  6. 2024A quantitative analysis of knowledge-learning preferences in large language models in molecular science
  7. 2024Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design
  8. 2025MolVision: Molecular Property Prediction with Vision Language Models