mlsanityData to 7 Oct 2026

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

arXiv 2305.14795 ↗First posted 24 May 2023cs.CLcited by 292

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model's related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark, MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (e.g., OpenAI GPT-3.5-turbo) and outperforms previous model editors by a large margin.

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

22 of 35 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 590
10 more
  1. 2021Editing Factual Knowledge in Language Modelscited by 548
  2. 2021Fast Model Editing at Scalecited by 491
  3. 2022Memory-Based Model Editing at Scalecited by 443
  4. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  5. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  6. 2020Modifying Memories in Transformer Modelscited by 222
  7. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  8. 2020Editable Neural Networkscited by 154
  9. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  10. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 118; 2025: 79; 2026: 71.

292 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  6. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  7. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  8. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  9. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  10. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  11. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  12. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
38 more
  1. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  2. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  3. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  6. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 92
  7. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  8. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  9. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  10. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  11. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  12. 2024Attention Heads of Large Language Models: A Surveycited by 65
  13. 2024Red Teaming Visual Language Modelscited by 62
  14. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  15. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  16. 2024Synthetic continued pretrainingcited by 56
  17. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  18. 2024A Unified Framework for Model Editingcited by 54
  19. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  20. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  21. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  22. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  23. 2023Can We Edit Multimodal Large Language Models?cited by 44
  24. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  25. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  26. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  27. 2024Knowledge Graph Enhanced Large Language Model Editingcited by 38
  28. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 38
  29. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  30. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  31. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  32. 2024Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Factscited by 36
  33. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 36
  34. 2024Uncovering Overfitting in Large Language Model Editingcited by 36
  35. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  36. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  37. 2023Evaluating the Reversal Curse in Model Editingcited by 34
  38. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33

50 most cited of 292 All 292 (JSON)