mlsanityData to 14 Oct 2026

Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

arXiv 2306.05783 ↗First posted 9 Jun 2023cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~urlhttps://github.com/MikeGu721/XiezhiBenchmark.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 39 of its 85 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

39 of 85 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023A Survey of Large Language Modelscited by 2,576
27 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  14. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  15. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  16. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  17. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  18. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  19. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  20. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  21. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  22. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  23. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  24. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  25. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  26. 2023Measuring Massive Multitask Chinese Understandingcited by 25
  27. 2023Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 29; 2025: 9; 2026: 5.

52 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  3. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  4. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  5. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  6. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  7. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  8. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 44
  9. 2024SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysiscited by 41
  10. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  11. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  12. 2024Small Language Model Can Self-correctcited by 23
38 more
  1. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  2. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Modelscited by 19
  3. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  4. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  5. 2024ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Modelscited by 13
  6. 2024CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Modelscited by 12
  7. 2023Evaluating the Generation Capabilities of Large Chinese Language Modelscited by 11
  8. 2024OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safetycited by 9
  9. 2024StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluationcited by 9
  10. 2024Efficiently Quantifying and Mitigating Ripple Effects in Model Editingcited by 6
  11. 2024AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behaviorcited by 6
  12. 2025MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environmentscited by 6
  13. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  14. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMscited by 5
  15. 2024CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimizationcited by 5
  16. 2024SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generationcited by 5
  17. 2024Leveraging Large Language Models for Enhanced Process Model Comprehensioncited by 5
  18. 2024PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Modelscited by 5
  19. 2024DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?cited by 4
  20. 2024LLM-GAN: Construct Generative Adversarial Network Through Large Language Models For Explainable Fake News Detectioncited by 4
  21. 2024DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicinecited by 4
  22. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 4
  23. 2024Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contaminationcited by 3
  24. 2025KaFT: Knowledge-aware Fine-tuning for Boosting LLMs' Domain-specific Question-Answering Performancecited by 3
  25. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 3
  26. 2023Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Modelscited by 2
  27. 2024FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Modelscited by 2
  28. 2025Identity Lock: Locking API Fine-tuned LLMs With Identity-based Wake Wordscited by 2
  29. 2025LITE: LLM-Impelled efficient Taxonomy Evaluationcited by 2
  30. 2024LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Modelscited by 1
  31. 2025SID: Benchmarking Guided Instruction Capabilities in STEM Education with a Socratic Interdisciplinary Dialogues Datasetcited by 1
  32. 2026Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discoverycited by 1
  33. 2024LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 0
  34. 2024Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Modelscited by 0
  35. 2024VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate Itcited by 0
  36. 2024KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Modelscited by 0
  37. 2025RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Modelcited by 0
  38. 2025BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Miningcited by 0

50 most cited of 52 All 52 (JSON)