# Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

arXiv: https://arxiv.org/abs/2306.05783 · HTML: https://mlsanity.com/paper/2306.05783/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 85 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)

## Built on by (most cited first, 30 of 52)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://mlsanity.com/paper/2309.04766/) (2023)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://mlsanity.com/paper/2307.15020/) (2023)
- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis](https://mlsanity.com/paper/2403.01976/) (2024)
- [SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding](https://mlsanity.com/paper/2408.15545/) (2024)
- [PRE: A Peer Review Based Large Language Model Evaluator](https://mlsanity.com/paper/2401.15641/) (2024)
- [Small Language Model Can Self-correct](https://mlsanity.com/paper/2401.07301/) (2024)
- [Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience](https://mlsanity.com/paper/2404.10315/) (2024)
- [SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models](https://mlsanity.com/paper/2406.09098/) (2024)
- [Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation](https://mlsanity.com/paper/2310.14424/) (2023)
- [SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models](https://mlsanity.com/paper/2503.13503/) (2025)
- [ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2407.05365/) (2024)
- [CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2407.02301/) (2024)
- [Evaluating the Generation Capabilities of Large Chinese Language Models](https://mlsanity.com/paper/2308.04823/) (2023)
- [OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety](https://mlsanity.com/paper/2403.12316/) (2024)
- [StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation](https://mlsanity.com/paper/2408.03281/) (2024)
- [Efficiently Quantifying and Mitigating Ripple Effects in Model Editing](https://mlsanity.com/paper/2403.07825/) (2024)
- [AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior](https://mlsanity.com/paper/2403.13433/) (2024)
- [MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments](https://mlsanity.com/paper/2501.01652/) (2025)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs](https://mlsanity.com/paper/2311.05374/) (2023)
- [CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization](https://mlsanity.com/paper/2405.04781/) (2024)
- [SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation](https://mlsanity.com/paper/2405.16552/) (2024)
- [Leveraging Large Language Models for Enhanced Process Model Comprehension](https://mlsanity.com/paper/2408.08892/) (2024)
- [PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models](https://mlsanity.com/paper/2412.06287/) (2024)

## Abstract (arXiv)

New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~urlhttps://github.com/MikeGu721/XiezhiBenchmark.