# Benchmarking Retrieval-Augmented Generation for Chemistry

arXiv: https://arxiv.org/abs/2505.07671 · HTML: https://mlsanity.com/paper/2505.07671/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://mlsanity.com/paper/2210.02627/) (2022)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [Reducing hallucination in structured outputs via Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.08189/) (2024)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2411.19443/) (2024)
- [ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction](https://mlsanity.com/paper/2310.13590/) (2023)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervision](https://mlsanity.com/paper/2307.01448/) (2023)

## Built on by (most cited first, 20 of 20)

- [TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models](https://mlsanity.com/paper/2601.18744/) (2026)
- [SUPERChem: A Multimodal Reasoning Benchmark in Chemistry](https://mlsanity.com/paper/2512.01274/) (2025)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings](https://mlsanity.com/paper/2508.01643/) (2025)
- [TritonDFT: Automating DFT with a Multi-Agent Framework](https://mlsanity.com/paper/2603.03372/) (2026)
- [MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models](https://mlsanity.com/paper/2608.23646/) (2026)
- [SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models](https://mlsanity.com/paper/2505.15094/) (2025)
- [Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research](https://mlsanity.com/paper/2506.12312/) (2025)
- [ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation](https://mlsanity.com/paper/2608.21504/) (2026)
- [RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval](https://mlsanity.com/paper/2506.08625/) (2025)
- [Language-Native Materials Processing Design by Lightly Structured Text Database and Reasoning Large Language Model](https://mlsanity.com/paper/2509.06093/) (2025)
- [ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks](https://mlsanity.com/paper/2510.18455/) (2025)
- [From Slides to Chatbots: Enhancing Large Language Models with University Course Materials](https://mlsanity.com/paper/2510.22272/) (2025)
- [Full-Stack Domain Enhancement for Combustion LLMs: Construction and Optimization](https://mlsanity.com/paper/2603.19268/) (2026)
- [PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design](https://mlsanity.com/paper/2605.26543/) (2026)
- [EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics](https://mlsanity.com/paper/2605.29394/) (2026)
- [MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry](https://mlsanity.com/paper/2606.05693/) (2026)
- [From Document Silos to Process Intelligence: A Multi-Layer Knowledge Graph for CMC Process Development](https://mlsanity.com/paper/2609.11493/) (2026)
- [UniK: Universal Knowledge Perception for Digital and Physical AI](https://mlsanity.com/paper/2609.23971/) (2026)
- [ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark](https://mlsanity.com/paper/2610.20012/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged as a powerful framework for enhancing large language models (LLMs) with external knowledge, particularly in scientific domains that demand specialized and dynamic information. Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high-quality, domain-specific corpora and well-curated evaluation benchmarks. In this work, we introduce ChemRAG-Bench, a comprehensive benchmark designed to systematically assess the effectiveness of RAG across a diverse set of chemistry-related tasks. The accompanying chemistry corpus integrates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. In addition, we present ChemRAG-Toolkit, a modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. Using ChemRAG-Toolkit, we demonstrate that RAG yields a substantial performance gain -- achieving an average relative improvement of 17.4% over direct inference methods. We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved passages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain. The code and data is available at https://chemrag.github.io.