# MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

arXiv: https://arxiv.org/abs/2403.08192 · HTML: https://mlsanity.com/paper/2403.08192/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 69 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)

## Built on by (most cited first, 22 of 22)

- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model](https://mlsanity.com/paper/2502.13449/) (2025)
- [MatTools: Benchmarking Large Language Models for Materials Science Tools](https://mlsanity.com/paper/2505.10852/) (2025)
- [FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models](https://mlsanity.com/paper/2508.01055/) (2025)
- [MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs](https://mlsanity.com/paper/2601.15279/) (2026)
- [How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehension](https://mlsanity.com/paper/2504.12314/) (2025)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [SubGDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learning](https://mlsanity.com/paper/2405.05665/) (2024)
- [BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions](https://mlsanity.com/paper/2506.05766/) (2025)
- [Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design](https://mlsanity.com/paper/2604.16279/) (2026)
- [MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models](https://mlsanity.com/paper/2608.23646/) (2026)
- [DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning](https://mlsanity.com/paper/2602.11089/) (2026)
- [MolGround: A Benchmark for Molecular Grounding](https://mlsanity.com/paper/2503.23668/) (2025)
- [Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understanding](https://mlsanity.com/paper/2602.02742/) (2026)
- [MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Revision](https://mlsanity.com/paper/2509.00063/) (2025)
- [MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding](https://mlsanity.com/paper/2607.01982/) (2026)
- [Divergence Decoding: Training-Free Capability Fusion](https://mlsanity.com/paper/2607.27248/) (2026)
- [ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning](https://mlsanity.com/paper/2610.17183/) (2026)
- [LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making](https://mlsanity.com/paper/2610.19780/) (2026)
- [ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark](https://mlsanity.com/paper/2610.20012/) (2026)

## Abstract (arXiv)

Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information, posing challenges to accurate molecular comprehension. Traditional evaluation metrics for generated content fail to assess a model's accuracy in molecular understanding. To rectify the absence of factual evaluation, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative molecular corpus. MoleculeQA is not only the first benchmark for molecular factual bias evaluation but also the largest QA dataset for molecular research. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific areas and pinpoints several particularly crucial factors for molecular understanding.