# Catch Me if You Search: When Contextual Web Search Results Affect the Detection of Hallucinations

arXiv: https://arxiv.org/abs/2504.01153 · HTML: https://mlsanity.com/paper/2504.01153/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 102 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation](https://mlsanity.com/paper/2310.15515/) (2023)
- [Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models](https://mlsanity.com/paper/2407.04121/) (2024)
- [Diving Deep into Modes of Fact Hallucinations in Dialogue Systems](https://mlsanity.com/paper/2301.04449/) (2023)
- [An Audit on the Perspectives and Challenges of Hallucinations in NLP](https://mlsanity.com/paper/2404.07461/) (2024)
- [Accurate and Nuanced Open-QA Evaluation Through Textual Entailment](https://mlsanity.com/paper/2405.16702/) (2024)
- [Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinations](https://mlsanity.com/paper/2404.03745/) (2024)
- [Study on the Helpfulness of Explainable Artificial Intelligence](https://mlsanity.com/paper/2410.11896/) (2024)

## Built on by (most cited first, 3 of 3)

- [Helicase: Uncertainty-Guided Supply Chain Knowledge Graph Construction with Autonomous Multi-Agent LLMs](https://mlsanity.com/paper/2605.26835/) (2026)
- [Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs](https://mlsanity.com/paper/2605.29928/) (2026)
- [AI Hallucination from Students' Perspective: A Thematic Analysis](https://mlsanity.com/paper/2602.17671/) (2026)

## Abstract (arXiv)

While we increasingly rely on large language models (LLMs) for various tasks, these models are known to produce inaccurate content or 'hallucinations' with potentially disastrous consequences. The recent integration of web search results into LLMs prompts the question of whether people utilize them to verify the generated content, thereby accurately detecting hallucinations. An online experiment (N=560) investigated how the provision of search results, either static (i.e., fixed search results provided by LLM) or dynamic (i.e., participant-led searches), affects participants' perceived accuracy of LLM-generated content (i.e., genuine, minor hallucination, major hallucination), self-confidence in accuracy ratings, as well as their overall evaluation of the LLM, as compared to the control condition (i.e., no search results). Results showed that participants in both static and dynamic conditions (vs. control) rated hallucinated content to be less accurate and perceived the LLM more negatively. However, those in the dynamic condition rated genuine content as more accurate and demonstrated greater overall self-confidence in their assessments than those in the static search or control conditions. We highlighted practical implications of incorporating web search functionality into LLMs in real-world contexts.