ScEdit: Script-based Assessment of Knowledge Editing
Not on the map. Why?
Abstract (arXiv)
Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark -- ScEdit (Script-based Knowledge Editing Benchmark) -- which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based ("What"-type question) evaluation to action-based ("How"-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 29 May 2025.
Builds on
32 of 51 references.
References · 32
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
- 2022Mass-Editing Memory in a Transformercited by 1,001
- 2022Memory-Based Model Editing at Scalecited by 447
- 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
- 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
- 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
- 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
- 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
- 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
20 more
- 2020Modifying Memories in Transformer Modelscited by 224
- 2023Knowledge Editing for Large Language Models: A Surveycited by 194
- 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
- 2020Editable Neural Networkscited by 155
- 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
- 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 116
- 2024Factuality of Large Language Models: A Surveycited by 66
- 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
- 2020A Dataset for Tracking Entities in Open Domain Procedural Textcited by 37
- 2024AKEW: Assessing Knowledge Editing in the Wildcited by 36
- 2021Goal-Oriented Script Constructioncited by 30
- 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
- 2024Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Taskscited by 24
- 2024Long-form evaluation of model editingcited by 23
- 2021What do Large Language Models Learn about Scripts?cited by 19
- 2025GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environmentscited by 19
- 2024Commonsense Knowledge Editing Based on Free-Text in LLMscited by 16
- 2024Event-level Knowledge Editingcited by 9
- 2024WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Modelscited by 9
- 2023Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customizationcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.