mlsanityData to 7 Oct 2026

RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature

arXiv 2512.23565 ↗First posted 29 Dec 2025cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

The integration of Multimodal Large Language Models (MLLMs) into chemistry promises to revolutionize scientific discovery, yet their ability to comprehend the dense, graphical language of reactions within authentic literature remains underexplored. Here, we introduce RxnBench, a multi-tiered benchmark designed to rigorously evaluate MLLMs on chemical reaction understanding from scientific PDFs. RxnBench comprises two tasks: Single-Figure QA (SF-QA), which tests fine-grained visual perception and mechanistic reasoning using 1,525 questions derived from 305 curated reaction schemes, and Full-Document QA (FD-QA), which challenges models to synthesize information from 108 articles, requiring cross-modal integration of text, schemes, and tables. Our evaluation of MLLMs reveals a critical capability gap: while models excel at extracting explicit text, they struggle with deep chemical logic and precise structural recognition. Notably, models with inference-time reasoning significantly outperform standard architectures, yet none achieve 50% accuracy on FD-QA. These findings underscore the urgent need for domain-specific visual encoders and stronger reasoning engines to advance autonomous AI chemists.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 38 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

17 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables13 Jun

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  7. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 373
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  9. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  10. 2017Learning to Plan Chemical Synthesescited by 175
  11. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  12. 2022SELFIES and the future of molecular string representationscited by 76
4 more
  1. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 76
  2. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  3. 2024MolParser: End-to-end Visual Recognition of Molecule Structures in the Wildcited by 17
  4. 2025Uni-Parser Technical Reportcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discoverycited by 7
  2. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3
  3. 2026From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
  2. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
  3. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
  4. 2025TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks
  5. 2025Towards Large-scale Chemical Reaction Image Parsing via a Multimodal Large Language Model
  6. 2026Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models
  7. 2024Probing the limitations of multimodal language models for chemistry and materials research
  8. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams