# WildSEEK: Evaluating Language Models for Information-Seeking

arXiv: https://arxiv.org/abs/2608.30683 · HTML: https://mlsanity.com/paper/2608.30683/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust](https://mlsanity.com/paper/2502.10844/) (2025)
- [COSMIC: Generalized Refusal Direction Identification in LLM Activations](https://mlsanity.com/paper/2506.00085/) (2025)
- [Citations and Trust in LLM Generated Responses](https://mlsanity.com/paper/2501.01303/) (2025)
- [LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing](https://mlsanity.com/paper/2511.03980/) (2025)
- [Loki: An Open-Source Tool for Fact Verification](https://mlsanity.com/paper/2410.01794/) (2024)
- [Recommenders with a mission: assessing diversity in newsrecommendations](https://mlsanity.com/paper/2012.10185/) (2020)
- [The AI Gap: How Socioeconomic Status Affects Language Technology Interactions](https://mlsanity.com/paper/2505.12158/) (2025)
- [Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking](https://mlsanity.com/paper/2402.05880/) (2024)
- [One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization](https://mlsanity.com/paper/2601.18572/) (2026)
- [Conversational AI increases political knowledge as effectively as self-directed internet search](https://mlsanity.com/paper/2509.05219/) (2025)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy](https://mlsanity.com/paper/2511.12920/) (2025)
- [Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs](https://mlsanity.com/paper/2508.05553/) (2025)
- [Large Language Models as Search Engines: Societal Challenges](https://mlsanity.com/paper/2512.08946/) (2025)
- [Overreliance on AI in Information-seeking from Video Content](https://mlsanity.com/paper/2603.19843/) (2026)
- [Structuring the Space of Perspectives](https://mlsanity.com/paper/2608.12113/) (2026)

## Abstract (arXiv)

Language models are increasingly mediating information access to end users, urging a systematic evaluation of their responses for a fair and reliable information ecosystem. Existing evaluations, however, are often topic-specific or synthetic, limiting their ability to capture the complexity of "in the wild" information-seeking queries and the risks present in model responses. To address this gap, we introduce WildSEEK, a manually annotated dataset of 3k information-seeking queries from real user interactions, and an evaluation framework for LLM-generated responses. WildSEEK includes annotations for risk-sensitive domains (e.g. health and financial information), and distinguishes factoid queries from analytical queries which seek responses beyond facts. We train classifiers on WildSEEK to analyze more than 1.8M realistic user queries. We find that over a third of information-seeking queries are high-risk and more often analytical. Our findings show that LLM responses fail more often in four criteria: sycophantic behavior, overreliance, a default US-centric perspective, and poor handling of vulnerable populations -- with failure rates being mostly higher for analytical queries. By providing methods to monitor the reliability, safety, and fairness of LLM behavior, our dataset and evaluation framework offer an empirical foundation for the broader question of how these systems should behave as they take on a growing role in information access.