mlsanityData to 14 Oct 2026

KoLA: Carefully Benchmarking World Knowledge of Large Language Models

arXiv 2306.09296 ↗First posted 15 Jun 2023cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For ability modeling, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For evaluation criteria, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate 28 open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 54 of its 85 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

54 of 85 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  15. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  16. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  17. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  18. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  20. 2018Assessing Generative Models via Precision and Recallcited by 406
  21. 2023Augmented Language Models: a Surveycited by 403
  22. 2023GPTScore: Evaluate as You Desirecited by 361
  23. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  24. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  25. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  26. 2015A note on the evaluation of generative modelscited by 292
  27. 2022RealTime QA: What's the Answer Right Now?cited by 244
  28. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  29. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  30. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 152
  31. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  32. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  33. 2020MAVEN: A Massive General Domain Event Detection Datasetcited by 104
  34. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  35. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 78
  36. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  37. 2022MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extractioncited by 66
  38. 2021CLEVE: Contrastive Pre-training for Event Extractioncited by 36

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 38; 2025: 16; 2026: 9.

85 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  3. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  4. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  5. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  8. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  9. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  10. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  11. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  12. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
38 more
  1. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  2. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  3. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  4. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  6. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 50
  7. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 50
  8. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  9. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 38
  10. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  11. 2024Benchmarking Agentic Workflow Generationcited by 37
  12. 2023Aligning Large Language Models with Human: A Surveycited by 32
  13. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  14. 2024On the Challenges and Opportunities in Generative AIcited by 31
  15. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  16. 2023UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generationcited by 22
  17. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  18. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  19. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 18
  20. 2024Causal Evaluation of Language Modelscited by 18
  21. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  22. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 17
  23. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  24. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  25. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 14
  26. 2024MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Casescited by 14
  27. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  28. 2024Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectivescited by 13
  29. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  30. 2024DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoningcited by 11
  31. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 10
  32. 2025LLMs as Repositories of Factual Knowledge: Limitations and Solutionscited by 10
  33. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 10
  34. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  35. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  36. 2024Towards a Holistic Evaluation of LLMs on Factual Knowledge Recallcited by 8
  37. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 8
  38. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7

50 most cited of 85 All 85 (JSON)