mlsanityData to 7 Oct 2026

MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model

arXiv 2503.04780 ↗First posted 23 Feb 2025cs.CL cs.AI physics.atom-phcited by 2

Not on the map. Why?

Abstract (arXiv)

Human expertise in chemistry and biomedicine relies on contextual molecular understanding, a capability that large language models (LLMs) can extend through fine-grained alignment between molecular structures and text. Recent multimodal learning advances focus on cross-modal alignment, but existing molecule-text models ignore complementary information in different molecular views and rely on single-view representations, limiting molecular understanding. Moreover, na"ive multi-view alignment strategies face two challenges: (1) separate aligned spaces with inconsistent mappings between molecule and text embeddings, and that (2) existing loss objectives fail to preserve complementary information for fine-grained alignment. This can limit the LLM's ability to fully understand the molecular properties. To address these issues, we propose MV-CLAM, a novel framework that aligns multi-view molecular representations into a unified textual space using a multi-query transformer (MQ-Former). Our approach ensures cross-view consistency while a token-level contrastive loss preserves diverse molecular features across textual queries. MV-CLAM enhances molecular reasoning, improving retrieval and captioning accuracy. The source code of MV-CLAM is available in https://github.com/sumin124/mv-clam.git.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  9. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  10. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 373
18 more
  1. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  2. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  3. 2022Translation between Molecules and Natural Languagecited by 173
  4. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  5. 2022Linearly Mapping from Image to Text Spacecited by 139
  6. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  7. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  8. 2020Molecule Attention Transformercited by 92
  9. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  10. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  11. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  12. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 65
  13. 2022Graph-based Molecular Representation Learningcited by 51
  14. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  15. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  16. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  17. 2023Diffusion-based Generative AI for Exploring Transition States from 2D Molecular Graphscited by 19
  18. 2024Learning Multi-view Molecular Representations with Structured and Unstructured Knowledgecited by 11

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.