mlsanityData to 14 Oct 2026

Translation between Molecules and Natural Language

arXiv 2204.11817 ↗First posted 25 Apr 2022cs.CL cs.AIcited by 177

Not on the map. Why?

Abstract (arXiv)

We present MolT5 - a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. MolT5 allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation (altogether: translation between molecules and language), which we explore for the first time. Since MolT5 pretrains models on single-modal data, it helps overcome the chemistry domain shortcoming of data scarcity. Furthermore, we consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation. Our results show that MolT5-based models are able to generate outputs, both molecules and captions, which in many cases are high quality.

Key facts. Cited by 177 papers in the map (data through 14 Oct 2026). Builds on 49 of its 91 references that are in the map. First posted on arXiv 25 Apr 2022.

Builds on

49 of 91 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
37 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2021Transformers in Vision: A Surveycited by 1,013
  8. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  13. 2019How multilingual is Multilingual BERT?cited by 645
  14. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  15. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  16. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  17. 2018Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Modelscited by 275
  18. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 267
  19. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  20. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  21. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  22. 2020X-Linear Attention Networks for Image Captioningcited by 198
  23. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  24. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 194
  25. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  26. 2019Image Captioning: Transforming Objects into Wordscited by 171
  27. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 94
  28. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  29. 2020Mapping the Space of Chemical Reactions Using Attention-Based Neural Networkscited by 70
  30. 2020Cross-media Structured Common Space for Multimedia Event Extractioncited by 61
  31. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  32. 2019A Survey on Biomedical Image Captioningcited by 30
  33. 2018Entity-aware Image Caption Generationcited by 26
  34. 2015EventNet: A Large Scale Structured Concept Library for Complex Event Detection in Videocited by 7
  35. 2021IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry Systemcited by 6
  36. 2021End-to-End Attention-based Image Captioningcited by 4
  37. 2021Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 36; 2024: 59; 2025: 40; 2026: 40.

177 papers.

Sort:

By area:

  1. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  2. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  3. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  4. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  5. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  6. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  7. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  8. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  9. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  10. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 93
  11. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 92
  12. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
38 more
  1. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  2. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  3. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  4. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  5. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  6. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 66
  7. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 61
  8. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  9. 2024BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuningcited by 54
  10. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 52
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  12. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  13. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  14. 2025El Agente: An Autonomous Agent for Quantum Chemistrycited by 44
  15. 2022A Systematic Survey of Chemical Pre-trained Modelscited by 34
  16. 2024ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedbackcited by 33
  17. 2023Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformerscited by 32
  18. 2023ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedbackcited by 31
  19. 2023Structured Chemistry Reasoning with Large Language Modelscited by 28
  20. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
  21. 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphscited by 25
  22. 2024Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generationcited by 25
  23. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  24. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  25. 2023Domain-Agnostic Molecular Generation with Chemical Feedbackcited by 22
  26. 2023Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materialscited by 22
  27. 2024LLaMo: Large Language Model-based Molecular Graph Assistantcited by 22
  28. 2024L+M-24: Building a Dataset for Language + Molecules @ ACL 2024cited by 21
  29. 2024LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Predictioncited by 21
  30. 2025Accelerating Scientific Discovery with Autonomous Goal-evolving Agentscited by 19
  31. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  32. 2024A quantitative analysis of knowledge-learning preferences in large language models in molecular sciencecited by 18
  33. 2024Large Language Models are In-Context Molecule Learnerscited by 18
  34. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  35. 2023Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Designcited by 17
  36. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17
  37. 2024LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learningcited by 17
  38. 2024HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignmentcited by 17

50 most cited of 177 All 177 (JSON)