mlsanityData to 14 Oct 2026

Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning

arXiv 2410.04223 ↗First posted 5 Oct 2024cs.LG physics.chem-ph q-bio.BMcited by 29

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

While large language models (LLMs) have integrated images, adapting them to graphs remains challenging, limiting their applications in materials and drug design. This difficulty stems from the need for coherent autoregressive generation across texts and graphs. To address this, we introduce Llamole, the first multimodal LLM capable of interleaved text and graph generation, enabling molecular inverse design with retrosynthetic planning. Llamole integrates a base LLM with the Graph Diffusion Transformer and Graph Neural Networks for multi-conditional molecular generation and reaction inference within texts, while the LLM, with enhanced molecular understanding, flexibly controls activation among the different graph modules. Additionally, Llamole integrates A* search with LLM-based cost functions for efficient retrosynthetic planning. We create benchmarking datasets and conduct extensive experiments to evaluate Llamole against in-context learning and supervised fine-tuning. Llamole significantly outperforms 14 adapted LLMs across 12 metrics for controllable molecular design and retrosynthetic planning.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 47 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

30 of 47 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2018How Powerful are Graph Neural Networks?cited by 4,393
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
18 more
  1. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  3. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 520
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  5. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 283
  6. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  7. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  8. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  9. 2022Sample Efficiency Matters: A Benchmark for Practical Molecular Optimizationcited by 155
  10. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  11. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  12. 2020Retro*: Learning Retrosynthetic Planning with Neural Guided A* Searchcited by 76
  13. 2022Graph Rationalization with Environment-based Augmentationscited by 70
  14. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  15. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  16. 2021Amortized Tree Generation for Bottom-up Synthesis Planning and Synthesizable Molecular Designcited by 44
  17. 2023Data-Centric Learning from Unlabeled Graphs with Diffusion Modelcited by 19
  18. 2023Semi-Supervised Graph Imbalanced Regressioncited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 13.

29 papers.

Sort:

By area:

  1. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  2. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  4. 2025LLM-Augmented Chemical Synthesis and Design Decision Programscited by 13
  5. 2025A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimizationcited by 13
  6. 2025AOT*: Efficient Synthesis Planning via LLM-Empowered AND-OR Tree Searchcited by 13
  7. 202534 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discoverycited by 9
  8. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  9. 2025MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generationcited by 9
  10. 2024Reflections from the 2024 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistrycited by 8
  11. 2025mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Moleculescited by 8
  12. 2025MolTextNet: A Two-Million Molecule-Text Dataset for Multimodal Molecular Learningcited by 8
17 more
  1. 2025Graph Diffusion Transformers are In-Context Molecular Designerscited by 7
  2. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  3. 2026Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesiscited by 6
  4. 2026Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Designcited by 4
  5. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructionscited by 3
  6. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  7. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluationcited by 2
  8. 2026SciDesignBench: Benchmarking and Improving Language Models for Scientific Inverse Designcited by 1
  9. 2026Project Ariadne: Prompt-Conditioned Route Generation for Synthesis Planningcited by 1
  10. 2026Closed-Loop Molecular Inverse Design with Semantic LLM Surrogatescited by 1
  11. 2024Exploiting Latent Linearity in LLMs Improves Explainable Molecular Representation Learningcited by 0
  12. 2025POINT2: A Polymer Informatics Training and Testing Databasecited by 0
  13. 2026Teaching and Evaluating LLMs to Reason About Polymer Design Related Taskscited by 0
  14. 2026MolLangData: A Large-Scale Dataset for Molecular Structure-Language Description via a Rule-Regularized Methodcited by 0
  15. 2026Reinforced Reasoning for End-to-End Retrosynthetic Planningcited by 0
  16. 2026CatalyticMLLM: A Graph-Text Multimodal Large Language Model for Catalytic Materialscited by 0
  17. 2026Molecular LLM Agents: From Architectural Design to Scientific Autonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Many-Shot In-Context Learning for Molecular Inverse Design
  2. 2025ChatMol: A Versatile Molecule Designer Based on the Numerically Enhanced Large Language Model
  3. 2026RetroAgent: Harnessing LLMs to Search Over Structured Memory for Agentic Retrosynthesis Planning
  4. 2025RetroDFM-R: Reasoning-Driven Retrosynthesis Prediction with Large Language Models via Reinforcement Learning
  5. 2024LLaMo: Large Language Model-based Molecular Graph Assistant
  6. 2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilization
  7. 2025LARC: Towards Human-level Constrained Retrosynthesis Planning through an Agentic Framework
  8. 2024Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design