# Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models

arXiv: https://arxiv.org/abs/2411.07140 · HTML: https://mlsanity.com/paper/2411.07140/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2024.

## Built on by (most cited first, 30 of 43)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning](https://mlsanity.com/paper/2505.11896/) (2025)
- [SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models](https://mlsanity.com/paper/2502.13059/) (2025)
- [When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.05690/) (2025)
- [Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory](https://mlsanity.com/paper/2505.15055/) (2025)
- [Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?](https://mlsanity.com/paper/2502.19361/) (2025)
- [CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward](https://mlsanity.com/paper/2508.03686/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2511.02805/) (2025)
- [Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs](https://mlsanity.com/paper/2503.05139/) (2025)
- [Semantic Energy: Detecting LLM Hallucination Beyond Entropy](https://mlsanity.com/paper/2508.14496/) (2025)
- [Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning](https://mlsanity.com/paper/2508.19598/) (2025)
- [Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Facts](https://mlsanity.com/paper/2510.21557/) (2025)
- [ERNIE 5.0 Technical Report](https://mlsanity.com/paper/2602.04705/) (2026)
- [Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models](https://mlsanity.com/paper/2412.15265/) (2024)
- [Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale](https://mlsanity.com/paper/2606.15079/) (2026)
- [dots.llm1 Technical Report](https://mlsanity.com/paper/2506.05767/) (2025)
- [MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs](https://mlsanity.com/paper/2510.22967/) (2025)
- [Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models](https://mlsanity.com/paper/2607.08186/) (2026)
- [Unbiased Evaluation of Large Language Models from a Causal Perspective](https://mlsanity.com/paper/2502.06655/) (2025)
- ["See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models](https://mlsanity.com/paper/2502.11718/) (2025)
- [Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering](https://mlsanity.com/paper/2505.16591/) (2025)
- [MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs](https://mlsanity.com/paper/2509.11662/) (2025)
- [Virtual Width Networks](https://mlsanity.com/paper/2511.11238/) (2025)
- [Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments](https://mlsanity.com/paper/2502.14275/) (2025)
- [Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap](https://mlsanity.com/paper/2508.18646/) (2025)
- [Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective](https://mlsanity.com/paper/2510.08800/) (2025)

## Abstract (arXiv)

New LLM evaluation benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where the reference answers are static and cannot be changed over time. Third, following SimpleQA, the questions and answers are very short, and the grading process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to better understand the Chinese factuality abilities of their models and facilitate the growth of foundation models.