# CANDY: Benchmarking LLMs' Limitations and Assistive Potential in Chinese Misinformation Fact-Checking

arXiv: https://arxiv.org/abs/2509.03957 · HTML: https://mlsanity.com/paper/2509.03957/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking](https://mlsanity.com/paper/1911.01214/) (2019)
- [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023)
- [Explainable Fake News Detection With Large Language Model via Defense Among Competing Wisdom](https://mlsanity.com/paper/2405.03371/) (2024)
- [Towards Human-centered Proactive Conversational Agents](https://mlsanity.com/paper/2404.12670/) (2024)
- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking](https://mlsanity.com/paper/2206.11863/) (2022)
- [Language Models Hallucinate, but May Excel at Fact Verification](https://mlsanity.com/paper/2310.14564/) (2023)
- [UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation](https://mlsanity.com/paper/2311.15296/) (2023)
- [Generative Large Language Models in Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2407.02351/) (2024)
- [From Chaos to Clarity: Claim Normalization to Empower Fact-Checking](https://mlsanity.com/paper/2310.14338/) (2023)
- [ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming](https://mlsanity.com/paper/2505.16667/) (2025)
- [LastingBench: Defend Benchmarks Against Knowledge Leakage](https://mlsanity.com/paper/2506.21614/) (2025)
- [Automatic Rumor Detection on Microblogs: A Survey](https://mlsanity.com/paper/1807.03505/) (2018)
- [CHECKED: Chinese COVID-19 Fake News Dataset](https://mlsanity.com/paper/2010.09029/) (2020)
- [Cross-lingual COVID-19 Fake News Detection](https://mlsanity.com/paper/2110.06495/) (2021)
- [How We Refute Claims: Automatic Fact-Checking through Flaw Identification and Explanation](https://mlsanity.com/paper/2401.15312/) (2024)
- [Explaining Veracity Predictions with Evidence Summarization: A Multi-Task Model Approach](https://mlsanity.com/paper/2402.06443/) (2024)
- [LTCR: Long-Text Chinese Rumor Detection Dataset](https://mlsanity.com/paper/2306.07201/) (2023)

## Built on by (most cited first, 1 of 1)

- [You Can't Fool Us: Understanding the Resilience of LLM-driven Agent Communities to Misinformation](https://mlsanity.com/paper/2605.17353/) (2026)

## Abstract (arXiv)

The effectiveness of large language models (LLMs) to fact-check misinformation remains uncertain, despite their growing use. To this end, we present CANDY, a benchmark designed to systematically evaluate the capabilities and limitations of LLMs in fact-checking Chinese misinformation. Specifically, we curate a carefully annotated dataset of ~20k instances. Our analysis shows that current LLMs exhibit limitations in generating accurate fact-checking conclusions, even when enhanced with chain-of-thought reasoning and few-shot prompting. To understand these limitations, we develop a taxonomy to categorize flawed LLM-generated explanations for their conclusions and identify factual fabrication as the most common failure mode. Although LLMs alone are unreliable for fact-checking, our findings indicate their considerable potential to augment human performance when deployed as assistive tools in scenarios. Our dataset and code can be accessed at https://github.com/SCUNLP/CANDY