# Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus

arXiv: https://arxiv.org/abs/2605.18490 · HTML: https://mlsanity.com/paper/2605.18490/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks](https://mlsanity.com/paper/2412.15605/) (2024)
- [WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems](https://mlsanity.com/paper/2605.07068/) (2026)

## Built on by (most cited first, 2 of 2)

- [Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation](https://mlsanity.com/paper/2607.04576/) (2026)
- [No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus](https://mlsanity.com/paper/2608.28875/) (2026)

## Abstract (arXiv)

We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: single-round Vector RAG and an LLM-compiled markdown wiki browsed by a tool-using agent. Both answered the same 13 questions over 24 papers with the same answer model, scored by two blinded LLM judges. The three preregistered predictions came out one weakly supported, one supported, and one refuted. The wiki scored much better at connecting findings across papers, but its organization advantage fell below the registered threshold once both judges were combined. RAG met the registered test on single-fact groundedness, though the result was judge-sensitive. The wiki was cheaper to build but spent about 21 times more LLM tokens per query, so no break-even point exists. Exploratory analyses bear on why such comparisons disagree. A decomposition-retrieval variant of RAG reduced most of the wiki's synthesis-score gap at lower token cost. The judges' rank agreement was near zero on holistic groundedness (rho = 0.04) against rho = 0.81 on the most concretely defined criterion. A post-hoc claim-level analysis of citation support was checked against two human annotators on 100 claims. Its scorer agreed with them on 50 to 54% of claims as first run and on 65 to 69% once a truncation error was corrected, short of the rule fixed in advance. On the annotators' labels, the analysis did not establish a citation-support advantage for the wiki. On one model and one small corpus, which system appears to win depends on the retrieval baseline, the scoring method, and the judge, so evaluations should report synthesis, citation support, and cost separately and check automated grounding scorers against human labels.