# Evaluating Commercial AI Chatbots as News Intermediaries

arXiv: https://arxiv.org/abs/2605.22785 · HTML: https://mlsanity.com/paper/2605.22785/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 71 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 5 of 5)

- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [AI use in American newspapers is widespread, uneven, and rarely disclosed](https://mlsanity.com/paper/2510.18774/) (2025)
- [Belief in the Machine: Investigating Epistemological Blind Spots of Language Models](https://mlsanity.com/paper/2410.21195/) (2024)
- [Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator](https://mlsanity.com/paper/2505.20738/) (2025)
- [When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation](https://mlsanity.com/paper/2509.26600/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systems](https://mlsanity.com/paper/2609.03425/) (2026)

## Abstract (arXiv)

AI chatbots are rapidly shaping how people encounter the news, yet no prior study has systematically measured how accurately these systems, with their proprietary search integrations and retrieval-synthesis pipelines, handle emerging facts across languages and regions. We present a 14-day (February 9-22, 2026) evaluation of six AI chatbots (Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5 and GPT-4o mini) on 2,100 factual questions derived from same-day BBC News reporting across six regional services (US & Canada, Arabic, Afrique, Hindi, Russian, Turkish). The best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier. The same systems, however, lose 11-13% under free-response evaluation, and 16-17% across the cohort. We further characterize three failure patterns. First, every model achieves its lowest accuracy on Hindi (79% vs. 89-91% elsewhere) and citations indicate an Anglophone retrieval bias (e.g., models answering Hindi queries cite English Wikipedia more than any Hindi outlet). Second, retrieval, not reasoning, failures drive over 70% of all errors. When models retrieve a correct source, they often extract the correct answer; the problem is to land on the right source in the first place. Third, models achieving 88-96% accuracy on well-formed questions drop to 19-70% when questions contain subtle false premises, with the most vulnerable model accepting fabricated facts 64% of the time. We also identify a detection-accuracy paradox: the best false-premise detector ranks second in adversarial accuracy (abstention rate), while a weaker detector ranks first, showing that premise detection and answer recovery are partially independent capabilities. Overall, these suggest that high accuracy can mask systematic regional inequity, near-total dependence on retrieval infrastructure, and vulnerability to imperfect queries real users pose.