# Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

arXiv: https://arxiv.org/abs/2511.18931 · HTML: https://mlsanity.com/paper/2511.18931/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025)
- [LLatrieval: LLM-Verified Retrieval for Verifiable Generation](https://mlsanity.com/paper/2311.07838/) (2023)
- [Dated Data: Tracing Knowledge Cutoffs in Large Language Models](https://mlsanity.com/paper/2403.12958/) (2024)
- [The Perils & Promises of Fact-checking with Large Language Models](https://mlsanity.com/paper/2310.13549/) (2023)
- [Are Large Language Models Temporally Grounded?](https://mlsanity.com/paper/2311.08398/) (2023)
- [Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration](https://mlsanity.com/paper/2405.16546/) (2024)
- [Alignment for Efficient Tool Calling of Large Language Models](https://mlsanity.com/paper/2503.06708/) (2025)
- [GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?](https://mlsanity.com/paper/2406.05606/) (2024)
- [Real-time Ad retrieval via LLM-generative Commercial Intention for Sponsored Search Advertising](https://mlsanity.com/paper/2504.01304/) (2025)
- [KnowRL: Teaching Language Models to Know What They Know](https://mlsanity.com/paper/2510.11407/) (2025)
- [DailyQA: A Benchmark to Evaluate Web Retrieval Augmented LLMs Based on Capturing Real-World Changes](https://mlsanity.com/paper/2505.17162/) (2025)
- [FAQ-Gen: An automated system to generate domain-specific FAQs to aid content comprehension](https://mlsanity.com/paper/2402.05812/) (2024)
- [Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge](https://mlsanity.com/paper/2506.18998/) (2025)

## Built on by (most cited first, 3 of 3)

- [What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)](https://mlsanity.com/paper/2608.06202/) (2026)
- [Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses](https://mlsanity.com/paper/2609.19244/) (2026)
- [AX is the New AEO](https://mlsanity.com/paper/2609.34951/) (2026)

## Abstract (arXiv)

Modern large language models increasingly integrate internal web-based retrieval to provide real-time answers, yet it remains unclear how effectively these systems identify information need, trigger retrieval, and use retrieved evidence. To understand these parameters better, we evaluate the necessity and effectiveness of internal web search through an external lens in closed-source LLMs, without access to model parameters or internal configuration. Our evaluation method comprises a static split of 783 temporally anchored factual queries answerable from pre-cutoff knowledge, designed to test whether retrieval is invoked based on factual uncertainty, and a dynamic split of 288 post-cutoff queries that require up-to-date information, designed to evaluate retrieval effectiveness under unavoidable information need. We experiment across four models across two model families and scales. Across models, enabling retrieval yields substantial accuracy gains on the static split, but systematically degrades confidence calibration. On the dynamic split, models frequently invoke retrieval yet remain below 70 percent accuracy, with failures dominated by query formulation and source selection errors rather than integration of retrieved content. While retrieval is inexpensive to invoke and can be selectively beneficial, repeated retrieval attempts rarely recover from early failures, and confidence becomes inflated once retrieval is available. Overall, internal web-based retrieval functions effectively as a low-latency verification mechanism, but falls short as a reliable IR pipeline, highlighting the need for improved retrieval triggering, query formulation, and evidence-aware confidence calibration in web-enabled LLMs.