mlsanityData to 7 Oct 2026

MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

arXiv 2403.08192 ↗First posted 13 Mar 2024cs.CL q-bio.BMcited by 19

Not on the map. Why?

Abstract (arXiv)

Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information, posing challenges to accurate molecular comprehension. Traditional evaluation metrics for generated content fail to assess a model's accuracy in molecular understanding. To rectify the absence of factual evaluation, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative molecular corpus. MoleculeQA is not only the first benchmark for molecular factual bias evaluation but also the largest QA dataset for molecular research. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific areas and pinpoints several particularly crucial factors for molecular understanding.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 40 of its 69 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

40 of 69 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
28 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  8. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  13. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  14. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  15. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  16. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  17. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  18. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  19. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  20. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  21. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  22. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  23. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  24. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 91
  25. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  26. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  27. 2023DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphscited by 40
  28. 2020Question-Driven Summarization of Answers to Consumer Health Questionscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  2. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 46
  3. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  4. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 16
  5. 2025MatTools: Benchmarking Large Language Models for Materials Science Toolscited by 11
  6. 2025FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Modelscited by 8
  7. 2025How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehensioncited by 6
  8. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  9. 2026MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphscited by 5
  10. 2024SubGDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learningcited by 4
  11. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  12. 2026Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Designcited by 3
7 more
  1. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3
  2. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  3. 2025MolGround: A Benchmark for Molecular Groundingcited by 1
  4. 2026Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understandingcited by 1
  5. 2025MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Revisioncited by 0
  6. 2026MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understandingcited by 0
  7. 2026Divergence Decoding: Training-Free Capability Fusioncited by 0