# Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation

arXiv: https://arxiv.org/abs/2412.14642 · HTML: https://mlsanity.com/paper/2412.14642/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [ChemLLM: A Chemical Large Language Model](https://mlsanity.com/paper/2402.06852/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area](https://mlsanity.com/paper/2408.07246/) (2024)
- [MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design](https://mlsanity.com/paper/2203.14500/) (2022)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [DrugAssist: A Large Language Model for Molecule Optimization](https://mlsanity.com/paper/2401.10334/) (2023)
- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation](https://mlsanity.com/paper/1905.13741/) (2019)
- [L+M-24: Building a Dataset for Language + Molecules @ ACL 2024](https://mlsanity.com/paper/2403.00791/) (2024)
- [Large Language Models are In-Context Molecule Learners](https://mlsanity.com/paper/2403.04197/) (2024)
- [HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignment](https://mlsanity.com/paper/2406.14021/) (2024)
- [SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration](https://mlsanity.com/paper/2409.02231/) (2024)
- [MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts](https://mlsanity.com/paper/2411.14721/) (2024)
- [Advancing biomolecular understanding and design following human instructions](https://mlsanity.com/paper/2410.07919/) (2024)

## Built on by (most cited first, 25 of 25)

- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [GeLLMO: Generalizing Large Language Models for Multi-property Molecule Optimization](https://mlsanity.com/paper/2502.13398/) (2025)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [Mol-R1: Towards Explicit Long-CoT Reasoning in Molecule Discovery](https://mlsanity.com/paper/2508.08401/) (2025)
- [MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs](https://mlsanity.com/paper/2601.15279/) (2026)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [GLProtein: Global-and-Local Structure Aware Protein Representation Learning](https://mlsanity.com/paper/2506.06294/) (2025)
- [HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens](https://mlsanity.com/paper/2512.15133/) (2025)
- [Enhancing Molecular Property Predictions by Learning from Bond Modelling and Interactions](https://mlsanity.com/paper/2603.00568/) (2026)
- [Geometric Flow Matching for Molecular Conformation Generation via Manifold Decomposition](https://mlsanity.com/paper/2605.25577/) (2026)
- [Chemical Chain-of-Thought Functions as a Hallucination-Prone Molecular Scratchpad](https://mlsanity.com/paper/2607.20935/) (2026)
- [From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models](https://mlsanity.com/paper/2606.03660/) (2026)
- [Mol-Debate: Multi-Agent Debate Improves Structural Reasoning in Molecular Design](https://mlsanity.com/paper/2604.20254/) (2026)
- [MolViBench: Evaluating LLMs on Molecular Vibe Coding](https://mlsanity.com/paper/2605.02351/) (2026)
- [MoleCode unlocks structural intelligence in large language models](https://mlsanity.com/paper/2605.16480/) (2026)
- [BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language](https://mlsanity.com/paper/2606.22138/) (2026)
- [Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysis](https://mlsanity.com/paper/2607.01800/) (2026)
- [Molecular LLM Agents: From Architectural Design to Scientific Autonomy](https://mlsanity.com/paper/2608.23104/) (2026)
- [SciMIF: Understanding Multimodal Instruction Following in Scientific Domains](https://mlsanity.com/paper/2608.25973/) (2026)
- [What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis](https://mlsanity.com/paper/2609.19212/) (2026)
- [MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design](https://mlsanity.com/paper/2609.27349/) (2026)
- [MARCO: Multi-Round Agentic Reinforcement for Conditional Molecular Optimization](https://mlsanity.com/paper/2609.36683/) (2026)
- [STAMP as a topology-aware molecular language for fragment-level learning and editing](https://mlsanity.com/paper/2610.18438/) (2026)
- [MolLean: A Process-Verifiable Language for Chemical Reasoning](https://mlsanity.com/paper/2610.20459/) (2026)

## Abstract (arXiv)

Recently, Large Language Models (LLMs) have demonstrated great potential in natural language-driven molecule discovery. However, existing datasets and benchmarks for molecule-text alignment are predominantly built on one-to-one mappings, measuring LLMs' ability to retrieve a single, pre-defined answer, rather than their creative potential to generate diverse, yet equally valid, molecular candidates. To address this critical gap, we propose Speak-to-Structure (S^2-Bench), the first benchmark to evaluate LLMs in open-domain natural language-driven molecule generation. S^2-Bench is specifically designed for one-to-many relationships, challenging LLMs to exhibit genuine molecular understanding and open-ended generation capabilities. Our benchmark includes three key tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and customized molecule generation (MolCustom), each probing a different aspect of molecule discovery. We also introduce OpenMolIns, a large-scale instruction tuning dataset that enables Llama3.1-8B to surpass the most powerful LLMs like GPT-4o and Claude-3.5 on S^2-Bench. Our comprehensive evaluation of 31 LLMs shifts the focus from simple pattern recall to realistic molecular design, paving the way for more capable LLMs in natural language-driven molecule discovery. Our codes and datasets are fully accessible through the Github Repository: https://github.com/phenixace/S2-TOMG-Bench and Huggingface Datasets: https://huggingface.co/datasets/phenixace/S2-TOMG-Bench.