mlsanityData to 7 Oct 2026

Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics

arXiv 2507.07155 ↗First posted 9 Jul 2025astro-ph.IM astro-ph.CO cs.AIcited by 1

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

also in 1 more
Abstract (arXiv)

We evaluate 9 Retrieval Augmented Generation (RAG) agent configurations on 105 Cosmology Question-Answer (QA) pairs that we built specifically for this purpose.The RAG configurations are manually evaluated by a human expert, that is, a total of 945 generated answers were assessed. We find that currently the best RAG agent configuration is with OpenAI embedding and generative model, yielding 91.4% accuracy. Using our human evaluation results we calibrate LLM-as-a-Judge (LLMaaJ) system which can be used as a robust proxy for human evaluation. These results allow us to systematically select the best RAG agent configuration for multi-agent system for autonomous scientific discovery in astrophysics (e.g., cmbagent presented in a companion paper) and provide us with an LLMaaJ system that can be scaled to thousands of cosmology QA pairs. We make our QA dataset, human evaluation results, RAG pipelines, and LLMaaJ system publicly available for further use by the astrophysics community.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 14 of its 26 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

14 of 26 references.

References · 14

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  6. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2019ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networkscited by 73
  9. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  10. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 37
  11. 2023Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomycited by 11
  12. 2024pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomycited by 6
2 more
  1. 2024What is the Role of Large Language Models in the Evolution of Astronomy Research?cited by 4
  2. 2024Designing an Evaluation Framework for Large Language Models in Astronomy Researchcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
  2. 2025SQuAI: Scientific Question-Answering with Multi-Agent Retrieval-Augmented Generation
  3. 2024A Collaborative Multi-Agent Approach to Retrieval-Augmented Generation Across Diverse Data
  4. 2026A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
  5. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  6. 2026Is Agentic RAG worth it? An experimental comparison of RAG approaches
  7. 2026MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation
  8. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Research