mlsanityData to 7 Oct 2026

Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy

arXiv 2511.12920 ↗First posted 17 Nov 2025cs.CL cs.AI cs.CYcited by 7

Not on the map. Why?

Abstract (arXiv)

Google Search increasingly surfaces AI-generated content through features like AI Overviews (AIO) and Featured Snippets (FS), which users frequently rely on despite having no control over their presentation. Through a systematic algorithm audit of 1,508 real baby care and pregnancy-related queries, we evaluate the quality and consistency of these information displays. Our robust evaluation framework assesses multiple quality dimensions, including answer consistency, relevance, presence of medical safeguards, source categories, and sentiment alignment. Our results reveal concerning gaps in information consistency, with information in AIO and FS displayed on the same search result page being inconsistent with each other in 33% of cases. Despite high relevance scores, both features critically lack medical safeguards (present in just 11% of AIO and 7% of FS responses). While health and wellness websites dominate source categories for both, AIO and FS, FS also often link to commercial sources. These findings have important implications for public health information access and demonstrate the need for stronger quality controls in AI-mediated health information. Our methodology provides a transferable framework for auditing AI systems across high-stakes domains where information quality directly impacts user well-being.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 16 of its 78 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

16 of 78 references.

References · 16

  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  4. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  5. 2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 43
  6. 2020ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Searchcited by 34
  7. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  8. 2019How to Ask Better Questions? A Large-Scale Multi-Domain Dataset for Rewriting Ill-Formed Questionscited by 12
  9. 2024Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authoritycited by 12
  10. 2024Evaluating Robustness of Generative Search Engine on Adversarial Factual Questionscited by 9
  11. 2023How does AI chat change search behaviors?cited by 5
  12. 2023The Evolution of Web Search User Interfaces -- An Archaeological Analysis of Google Search Engine Result Pagescited by 3
4 more
  1. 2025Human Trust in AI Search: A Large-Scale Experimentcited by 3
  2. 2025Reduced AI Acceptance After the Generative AI Boom: Evidence From a Two-Wave Survey Studycited by 2
  3. 2019Featured Snippets Results in Google Web Search: An Exploratory Studycited by 1
  4. 2021Searching for Representation: A sociotechnical audit of googling for members of U.S. Congresscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.