mlsanityData to 7 Oct 2026

Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers

arXiv 2602.22221 ↗First posted 15 Dec 2025cs.IR cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Search engines and AI-powered systems increasingly mediate access to factual information, yet their reliability remains difficult to evaluate in realistic information-seeking settings. We study this problem in the Chinese web ecosystem by constructing a query-based fact-checking dataset from real Chinese search logs and comparing nine systems across traditional search engines, standalone large language models, and search-integrated AI Overviews. Focusing on factual Chinese-language factual Yes/No questions, we evaluate whether systems provide correct, incorrect, or uncertain decisions against evidence-derived ground truth. We find that systems are similarly accurate when they provide definitive answers, but differ sharply in how often they do so. Conditional accuracy ranges from 73.2% to 78.9%, yet search engines answer definitively on over 83% of queries, while Qwen-Max does so on fewer than half. We also find a consistent polarity gap: all systems perform better on yes-labeled queries than on no-labeled queries. We also use Baidu Index data to identify Chinese provinces with higher health-related search attention, which may indicate greater potential exposure to misinformation. Overall, our results show that reliability depends not only on whether systems are correct when they answer, but also on how often they answer, how they handle negative claims, and where information demand may increase exposure risks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

15 of 30 references.

References · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  5. 2021A Survey on Automated Fact-Checkingcited by 376
  6. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  7. 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
  8. 2019GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verificationcited by 100
  9. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  10. 2022CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checkingcited by 37
  11. 2024CFEVER: A Chinese Fact Extraction and VERification Datasetcited by 5
  12. 2024Evaluating Search Engines and Large Language Models for Answering Health Questionscited by 5
3 more
  1. 2024Trusting the Search: Unraveling Human Trust in Health Information from Google and ChatGPTcited by 3
  2. 2024The Impact of Snippet Reliability on Misinformation in Online Health Searchcited by 2
  3. 2025Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Searchcited by 2

Built on by

Not cited yet.