mlsanityData to 7 Oct 2026

ALCUNA: Large Language Models Meet New Knowledge

arXiv 2310.14820 ↗First posted 23 Oct 2023cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

With the rapid development of NLP, large-scale language models (LLMs) excel in various tasks across multiple domains now. However, existing benchmarks may not adequately measure these models' capabilities, especially when faced with new knowledge. In this paper, we address the lack of benchmarks to evaluate LLMs' ability to handle new knowledge, an important and challenging aspect in the rapidly evolving world. We propose an approach called KnowGen that generates new knowledge by altering existing entity attributes and relationships, resulting in artificial entities that are distinct from real-world entities. With KnowGen, we introduce a benchmark named ALCUNA to assess LLMs' abilities in knowledge understanding, differentiation, and association. We benchmark several LLMs, reveals that their performance in face of new knowledge is not satisfactory, particularly in reasoning between new and internal knowledge. We also explore the impact of entity similarity on the model's understanding of entity knowledge and the influence of contextual entities. We appeal to the need for caution when using LLMs in new scenarios or with new knowledge, and hope that our benchmarks can help drive the development of LLMs in face of new knowledge.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

26 of 37 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
14 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  8. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  9. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  10. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  11. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
  12. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  13. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  14. 2022The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 13; 2026: 6.

35 papers.

Sort:

By area:

  1. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  2. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  3. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  4. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  5. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  6. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  7. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 48
  8. 2024Can AI Assistants Know What They Don't Know?cited by 47
  9. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 37
  10. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  11. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  12. 2024DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMscited by 23
23 more
  1. 2024Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Modelscited by 14
  2. 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attackscited by 14
  3. 2025Refine Knowledge of Large Language Models via Adaptive Contrastive Learningcited by 14
  4. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 11
  5. 2025LLMs as Repositories of Factual Knowledge: Limitations and Solutionscited by 10
  6. 2026Over-Searching in Search-Augmented Large Language Modelscited by 10
  7. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 9
  8. 2025How new data permeates LLM knowledge and how to dilute itcited by 8
  9. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Modelscited by 8
  10. 2025Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifiescited by 8
  11. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 7
  12. 2024Learning and Unlearning of Fabricated Knowledge in Language Modelscited by 4
  13. 2025Delusions of Large Language Modelscited by 4
  14. 2025A Comprehensive Taxonomy of Negation for NLP and Neural Retrieverscited by 4
  15. 2024FAME: Towards Factual Multi-Task Model Editingcited by 3
  16. 2025AutoEvoEval: An Automated Framework for Evolving Close-Ended LLM Evaluation Datacited by 3
  17. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scalingcited by 3
  18. 2024Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Modelscited by 2
  19. 2025ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generationcited by 1
  20. 2025ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Modelscited by 1
  21. 2026PhantomBench: Benchmarking the Non-existential Threat of Language Modelscited by 1
  22. 2025Noise or Nuance: An Investigation Into Useful Information and Filtering For LLM Driven AKBCcited by 0
  23. 2026InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysiscited by 0