mlsanityData to 14 Oct 2026

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

arXiv 2608.16096 ↗First posted 17 Aug 2026cs.IR cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a buyer needs before a published number can be used: whether the retrieval backbone may be deployed commercially, and what it costs to build. On licensing: the field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0. Of the four leading MuSiQue systems we audit (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none says so. On performance: we measure thirteen embedders from eight makers on one identical MuSiQue harness with bootstrap confidence intervals throughout. Until mid-2026 there was a real commercial tax: the best commercially-licensed embedder trailed the anchor by 2.31 Recall@5 points (95% CI [0.91, 3.71], p=0.001). NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, has closed it: +0.24 at Recall@5 (95% CI [-0.94, +1.43], p=0.69), -0.58 at Recall@10 (p=0.28). It matches the anchor, does not beat it, and is the only entrant that is commercially licensed, free to self-host, and indistinguishable from the anchor; every other entrant meeting the first two conditions sits 5.2 to 14.6 points below. The durable finding is the paid-versus-free divide: API embedders charge per token on every re-index, self-hosted ones charge nothing. On cost: three of five audited systems (adding Microsoft's GraphRAG) do not disclose indexing cost, and the only published GraphRAG dollar figures span 11x inside one third-party paper (USD 2.30 vs USD 24.94 to index a 5.64 MB corpus once); extrapolated to 1 TB that undisclosed choice separates roughly USD 428K from 4.6M. Our cost model keeps one-time embedding apart from recurring answering: at 1 TB, embedding sits 7.5x-900x below graph construction, and a year of answering at 10,000 queries/day sits 350x or more below it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

9 of 25 references.

References · 9

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  5. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  6. 2025KET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAGcited by 29
  7. 2025PropRAG: Guiding Retrieval with Beam Search over Proposition Pathscited by 23
  8. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluationcited by 8
  9. 2026SAG: SQL-Retrieval Augmented Generation with Query-Time Dynamic Hyperedgescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
  2. 2026Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems
  3. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmark
  4. 2024NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval
  5. 2026Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families
  6. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
  7. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems
  8. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch