mlsanityData to 7 Oct 2026

Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus

arXiv 2605.18490 ↗First posted 18 May 2026cs.CL cs.IRcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: single-round Vector RAG and an LLM-compiled markdown wiki browsed by a tool-using agent. Both answered the same 13 questions over 24 papers with the same answer model, scored by two blinded LLM judges. The three preregistered predictions came out one weakly supported, one supported, and one refuted. The wiki scored much better at connecting findings across papers, but its organization advantage fell below the registered threshold once both judges were combined. RAG met the registered test on single-fact groundedness, though the result was judge-sensitive. The wiki was cheaper to build but spent about 21 times more LLM tokens per query, so no break-even point exists. Exploratory analyses bear on why such comparisons disagree. A decomposition-retrieval variant of RAG reduced most of the wiki's synthesis-score gap at lower token cost. The judges' rank agreement was near zero on holistic groundedness (rho = 0.04) against rho = 0.81 on the most concretely defined criterion. A post-hoc claim-level analysis of citation support was checked against two human annotators on 100 claims. Its scorer agreed with them on 50 to 54% of claims as first run and on 65 to 69% once a truncation error was corrected, short of the rule fixed in advance. On the annotators' labels, the analysis did not establish a citation-support advantage for the wiki. On one model and one small corpus, which system appears to win depends on the retrieval baseline, the scoring method, and the judge, so evaluations should report synthesis, citation support, and cost separately and check automated grounding scorers against human labels.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 25 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

18 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan

Other references · 17

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  12. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
5 more
  1. 2023Active Retrieval Augmented Generationcited by 506
  2. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  3. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  4. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  5. 2026WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablationcited by 0
  2. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation
  2. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
  3. 2026A Fixed-Budget, Cluster-Aware Standard for LLM-as-a-Judge Evaluation: A Multi-Hop RAG Stress Test
  4. 2025Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
  5. 2024Evaluating Students' Open-ended Written Responses with LLMs: Using the RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large
  6. 2025DRAGOn: Designing RAG On Periodically Updated Corpus
  7. 2025Never Come Up Empty: Adaptive HyDE Retrieval for Improving LLM Developer Support
  8. 2024Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations