mlsanityData to 14 Oct 2026

AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

arXiv 2609.16519 ↗First posted 15 Sep 2026cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that provide natural language access to scientific knowledge and research workflows. Researchers are exploring the viability of these systems as natural language interfaces for document search and for generating analysis code and pipeline components. At the same time, concerns about data privacy and control over research infrastructure have motivated interest in open-weight models and open-source deployments hosted within research institutions. In astronomy, this development follows a long history of computational infrastructure development, from archival databases and Structured Query Language (SQL)-based systems to large language model (LLM)-assisted research tools. This paper presents a domain-expert evaluation of faithfulness for AquiLLM, an open-weight, offline RAG-LLM platform designed to support scientific research groups in the use and preservation of tacit and formal knowledge. We define faithfulness as the extent to which generated responses remain grounded in retrieved scientific context without unsupported claims or omissions. We report results from an astronomy case study evaluating AquiLLM across retrieval and scientific analysis tasks. AquiLLM performs most reliably on explicit retrieval-oriented questions grounded in the RAG collection, while faithfulness degrades for queries requiring synthesis or ambiguity resolution. These results highlight both the promise and limitations of open-weight RAG-LLM systems for scientific research and demonstrate the importance of domain-expert evaluation beyond standard benchmark leaderboards.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 24 of its 59 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

24 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation10 Mar

Other references · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  7. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  8. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  9. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  10. 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41
  11. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  12. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
11 more
  1. 2024GalaxiesML: a dataset of galaxy images, photometry, redshifts, and structural parameters for machine learningcited by 6
  2. 2024Understanding Galaxy Morphology Evolution Through Cosmic Time via Redshift Conditioned Diffusion Modelscited by 4
  3. 2024Using different sources of ground truths and transfer learning to improve the generalization of photometric redshift estimationcited by 3
  4. 2022Elements of effective machine learning datasets in astronomycited by 2
  5. 2024Using Galaxy Evolution as Source of Physics-Based Ground Truth for Generative Modelscited by 2
  6. 2025AquiLLM: a RAG Tool for Capturing Tacit Knowledge in Research Groupscited by 2
  7. 2025Multi-Modal Masked Autoencoders for Learning Image-Spectrum Associations for Galaxy Evolution and Cosmologycited by 2
  8. 2025Large language models in materials science and the need for open-source approachescited by 2
  9. 2026Improving Generalization and Uncertainty Quantification of Photometric Redshift Modelscited by 2
  10. 2026Combining datasets with different ground truths using Low-Rank Adaptation to generalize image-based CNN models for photometric redshift predictioncited by 1
  11. 2026AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groupscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Designing an Evaluation Framework for Large Language Models in Astronomy Research
  2. 2026Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy
  3. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
  4. 2026DS@GT ARC at LongEval: Citation Integrity and Factual Grounding in Scientific QA
  5. 2024C-FedRAG: A Confidential Federated Retrieval-Augmented Generation System
  6. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
  7. 2025FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation
  8. 2025SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA