# MolViBench: Evaluating LLMs on Molecular Vibe Coding

arXiv: https://arxiv.org/abs/2605.02351 · HTML: https://mlsanity.com/paper/2605.02351/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 18 of 18)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation](https://mlsanity.com/paper/2412.14642/) (2024)
- [CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics](https://mlsanity.com/paper/2508.18124/) (2025)

## Built on by (most cited first, 1 of 1)

- [Molecular LLM Agents: From Architectural Design to Scientific Autonomy](https://mlsanity.com/paper/2608.23104/) (2026)

## Abstract (arXiv)

Molecular Vibe Coding, a paradigm where chemists interact with LLMs to generate executable programs for molecular tasks, has emerged as a flexible alternative to chemical agents with predefined tools, enabling chemists to express arbitrarily complex, customized workflows. Unlike general coding tasks, molecular coding imposes a distinctive challenge that LLMs should jointly equip programming, molecular understanding, and domain-specific reasoning capabilities. However, existing benchmarks remain disconnected. General code generation benchmarks such as HumanEval and SWE-bench require no chemistry knowledge, while chemistry-focused benchmarks such as S^2-Bench and ChemCoTBench evaluate knowledge recall or property prediction rather than executable code generation. To bridge this gap, we introduce MolViBench, the first benchmark tailored for Molecular Vibe Coding. MolViBench comprises 358 curated tasks across five cognitive levels, ranging from single-API recall to end-to-end virtual screening pipeline design, spanning 12 real-world drug discovery workflows. To rigorously assess generated code, we also propose a multi-layered evaluation framework that combines type-aware output comparison and AST-based API-semantic fallback analysis, which jointly measures executability and chemical correctness. We systematically evaluate 9 frontier coding LLMs and compare three real-world Molecular Vibe Coding paradigms, providing a practical and fine-grained testbed for diagnosing LLMs' coding capabilities in AI-accelerated molecular discovery.