mlsanityData to 14 Oct 2026

Towards 3D Molecule-Text Interpretation in Language Models

arXiv 2401.13923 ↗First posted 25 Jan 2024cs.LG cs.IR q-bio.BMcited by 66

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We release our codes and datasets at https://github.com/lsh0520/3D-MoLM.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

38 of 59 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
26 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2022Galactica: A Large Language Model for Sciencecited by 744
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  12. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  13. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  14. 2022Translation between Molecules and Natural Languagecited by 177
  15. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  16. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 166
  17. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  18. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  19. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  20. 2022Let Invariant Rationale Discovery Inspire Graph Contrastive Learningcited by 57
  21. 2022Root-aligned SMILES: A Tight Representation for Chemical Reaction Predictioncited by 51
  22. 2023Rethinking Tokenizer and Decoder in Masked Graph Modeling for Moleculescited by 49
  23. 2021Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphscited by 31
  24. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  25. 2023Invariant Collaborative Filtering to Popularity Distribution Shiftcited by 23
  26. 2023Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+cited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 28; 2025: 24; 2026: 14.

66 papers.

Sort:

By area:

  1. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  2. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  3. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  4. 2024ProtT3: Protein-to-Text Generation for Text-based Protein Understandingcited by 25
  5. 2024Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generationcited by 25
  6. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  7. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  8. 2024MolTC: Towards Molecular Relational Modeling In Language Modelscited by 22
  9. 2024Prospect Personalized Recommendation on Large Language Model-based Agent Platformcited by 19
  10. 2024ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretrainingcited by 19
  11. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17
  12. 2024HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignmentcited by 17
38 more
  1. 2025NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generationcited by 17
  2. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 17
  3. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  4. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
  5. 2024PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomescited by 14
  6. 2025MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMscited by 12
  7. 2024Learning Multi-view Molecular Representations with Structured and Unstructured Knowledgecited by 11
  8. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  9. 20243D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modelingcited by 9
  10. 2024Advancing Molecular Graph-Text Pre-training via Fine-grained Alignmentcited by 9
  11. 2025OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discoverycited by 8
  12. 2025mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Moleculescited by 8
  13. 2024MolBind: Multimodal Alignment of Language, Molecules, and Proteinscited by 7
  14. 2025Language-Enhanced Representation Learning for Single-Cell Transcriptomicscited by 7
  15. 2025KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledgecited by 7
  16. 2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilizationcited by 6
  17. 2025How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehensioncited by 6
  18. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  19. 2024MolX: Enhancing Large Language Models for Molecular Understanding With A Multi-Modal Extensioncited by 5
  20. 2024Text-guided Diffusion Model for 3D Molecule Generationcited by 5
  21. 2025Bridging Molecular Graphs and Large Language Modelscited by 5
  22. 2024Bridging Text and Molecule: A Survey on Multimodal Frameworks for Moleculecited by 4
  23. 2024Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrievalcited by 3
  24. 2025Omni-Mol: Multitask Molecular Model for Any-to-any Modalitiescited by 3
  25. 2025MolEdit: Knowledge Editing for Multimodal Molecule Language Modelscited by 3
  26. 2024Atomas: Hierarchical Alignment on Molecule-Text for Unified Molecule Understanding and Generationcited by 2
  27. 2024Exploring Hierarchical Molecular Graph Representation in Multimodal LLMscited by 2
  28. 2024OneProt: Towards Multi-Modal Protein Foundation Modelscited by 2
  29. 2024GeomCLIP: Contrastive Geometry-Text Pre-training for Moleculescited by 2
  30. 2025MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Modelcited by 2
  31. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  32. 2025A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunitiescited by 2
  33. 2026Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learningcited by 2
  34. 2025PRING: Rethinking Protein-Protein Interaction Prediction from Pairs to Graphscited by 1
  35. 2025MoRA: On-the-fly Molecule-aware Low-Rank Adaptation Framework for LLM-based Multi-Modal Molecular Assistantcited by 1
  36. 2025EnzyControl: Adding Functional and Substrate-Specific Control for Enzyme Backbone Generationcited by 1
  37. 2025RTMol: Rethinking Molecule-text Alignment in a Round-trip Viewcited by 1
  38. 2026Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understandingcited by 1

50 most cited of 66 All 66 (JSON)