mlsanityData to 7 Oct 2026

Evaluating Verifiability in Generative Search Engines

arXiv 2304.09848 ↗First posted 19 Apr 2023cs.CL cs.IRcited by 297

Not on the map. Why?

Cited by 48 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative search engines directly generate responses to user queries, along with in-line citations. A prerequisite trait of a trustworthy generative search engine is verifiability, i.e., systems should cite comprehensively (high citation recall; all statements are fully supported by citations) and accurately (high citation precision; every cite supports its associated statement). We conduct human evaluation to audit four popular generative search engines -- Bing Chat, NeevaAI, perplexity.ai, and YouChat -- across a diverse set of queries from a variety of sources (e.g., historical Google user queries, dynamically-collected open-ended questions on Reddit, etc.). We find that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations: on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence. We believe that these results are concerningly low for systems that may serve as a primary tool for information-seeking users, especially given their facade of trustworthiness. We hope that our results further motivate the development of trustworthy generative search engines and help researchers and users better understand the shortcomings of existing commercial systems.

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

15 of 28 references.

References · 15

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  6. 2019ELI5: Long Form Question Answeringcited by 567
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  10. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  11. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
3 more
  1. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  2. 2019Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claimscited by 73
  3. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 66; 2024: 84; 2025: 60; 2026: 87.

297 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  8. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  10. 2023Conformal Language Modelingcited by 163
  11. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  12. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
38 more
  1. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  2. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  3. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  4. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  5. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  6. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 111
  7. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  8. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  9. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  10. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  11. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  12. 2023A Survey of Large Language Models Attributioncited by 71
  13. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  14. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  15. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  16. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  17. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 61
  18. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutionscited by 59
  19. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  20. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 57
  21. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  22. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  23. 2023GEO: Generative Engine Optimizationcited by 53
  24. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  25. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  26. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  27. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  28. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  29. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  30. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  31. 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversationscited by 43
  32. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  33. 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 42
  34. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  35. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 40
  36. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  37. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  38. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36

50 most cited of 297 All 297 (JSON)