mlsanityData to 7 Oct 2026

STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases

arXiv 2404.13207 ↗First posted 19 Apr 2024cs.IR cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Answering real-world complex queries, such as complex product search, often requires accurate retrieval from semi-structured knowledge bases that involve blend of unstructured (e.g., textual descriptions of products) and structured (e.g., entity relations of products) information. However, many previous works studied textual and relational retrieval tasks as separate topics. To address the gap, we develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties, together with their ground-truth answers (items). We conduct rigorous human evaluation to validate the quality of our synthesized queries. We further enhance the benchmark with high-quality human-generated queries to provide an authentic reference. STARK serves as a comprehensive testbed for evaluating the performance of retrieval systems driven by large language models (LLMs). Our experiments suggest that STARK presents significant challenges to the current retrieval and LLM systems, highlighting the need for more capable semi-structured retrieval systems. The benchmark data and code are available on https://github.com/snap-stanford/STaRK.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 37 of its 87 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

37 of 87 references.

References · 37

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
25 more
  1. 2015Image-based Recommendations on Styles and Substitutescited by 702
  2. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  3. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  4. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  5. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  6. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  7. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  8. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  10. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  11. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  12. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  13. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  14. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  15. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  16. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  17. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  18. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  19. 2015Large-scale Simple Question Answering with Memory Networkscited by 127
  20. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  21. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 76
  22. 2014Question Answering with Subgraph Embeddingscited by 56
  23. 2016TabMCQ: A Dataset of General Knowledge Tables and Multiple-choice Questionscited by 18
  24. 2019Querying Knowledge via Multi-Hop English Questionscited by 2
  25. 2017Knowledge Rich Natural Language Queries over Structured Biological Databasescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 23; 2026: 18.

53 papers.

Sort:

By area:

  1. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  2. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  4. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  5. 2024HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Basescited by 26
  6. 2024Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrievalcited by 24
  7. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  8. 2024KGARevion: An AI Agent for Knowledge-Intensive Biomedical QAcited by 20
  9. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Surveycited by 18
  10. 2024CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Eracited by 13
  11. 2024RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplementscited by 11
  12. 2025BYOKG-RAG: Multi-Strategy Graph Retrieval for Knowledge Graph Question Answeringcited by 10
38 more
  1. 2025Mixture of Structural-and-Textual Retrieval over Text-rich Graph Knowledge Basescited by 9
  2. 2025Align-GRAG: Anchor and Rationale Guided Dual Alignment for Graph Retrieval-Augmented Generationcited by 9
  3. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
  4. 2024RAG-based Question Answering over Heterogeneous Data and Textcited by 7
  5. 2024RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicinecited by 6
  6. 2025GraphRAFT: Retrieval Augmented Fine-Tuning for Knowledge Graphs on Graph Databasescited by 5
  7. 2024Harnessing the Power of Semi-Structured Knowledge and LLMs with Triplet-Based Prefiltering for Question Answeringcited by 4
  8. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  9. 2025Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agentscited by 4
  10. 2024Multi-Field Adaptive Retrievalcited by 3
  11. 2025FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQLcited by 3
  12. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuningcited by 3
  13. 2026Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrievalcited by 3
  14. 2026KV-Skill: Forging Expertise in the Model's Native Languagecited by 3
  15. 2025Querying Databases with Function Callingcited by 2
  16. 2025Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledgecited by 2
  17. 2025Knowledge-Augmented Long-CoT Generation for Complex Biomolecular Reasoningcited by 2
  18. 2026Autonomous Knowledge Graph Exploration with Adaptive Breadth-Depth Retrievalcited by 2
  19. 2026Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphscited by 2
  20. 2026RL-Index: Reinforcement Learning for Retrieval Index Reasoningcited by 2
  21. 2024Natural Language Interaction with a Household Electricity Knowledge-based Digital Twincited by 1
  22. 2025SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMscited by 1
  23. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  24. 2025SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrievalcited by 1
  25. 2026Textual Equilibrium Propagation for Deep Compound AI Systemscited by 1
  26. 2026Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimizationcited by 1
  27. 2025A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Modelscited by 0
  28. 2025How good are LLMs at Retrieving Documents in a Specific Domain?cited by 0
  29. 2025Rationale-Augmented Retrieval with Constrained LLM Re-Ranking for Task Discoverycited by 0
  30. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  31. 2026SAGE: Structure Aware Graph Expansion for Retrieval of Heterogeneous Datacited by 0
  32. 2025Comparative Analysis of Neural Retriever-Reranker Pipelines for Retrieval-Augmented Generation over Knowledge Graphs in E-commerce Applicationscited by 0
  33. 2026DSL-R1: From SQL to DSL for Training Retrieval Agents across Structured and Unstructured Data with Reinforcement Learningcited by 0
  34. 2026Ex-GraphRAG: Interpretable Evidence Routing for Graph-Augmented LLMscited by 0
  35. 2026SeedER: Seed-Expand-Retrieve for Efficient Knowledge Graph Retrievalcited by 0
  36. 2026Executable Schema Contracts: From Automatic Ingestion to Multi-Source Retrievalcited by 0
  37. 2026GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphscited by 0
  38. 2026Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuningcited by 0

50 most cited of 53 All 53 (JSON)