# Model Editing for LLMs4Code: How Far are We?

arXiv: https://arxiv.org/abs/2411.06638 · HTML: https://mlsanity.com/paper/2411.06638/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 30 of 50)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)

## Built on by (most cited first, 10 of 10)

- [CODEMENV: Benchmarking Large Language Models on Code Migration](https://mlsanity.com/paper/2506.00894/) (2025)
- [CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale](https://mlsanity.com/paper/2502.16645/) (2025)
- [Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience](https://mlsanity.com/paper/2508.19292/) (2025)
- [LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment](https://mlsanity.com/paper/2412.18135/) (2024)
- [Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering](https://mlsanity.com/paper/2508.11824/) (2025)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [Understanding Robustness of Model Editing in Code LLMs](https://mlsanity.com/paper/2511.03182/) (2025)
- [Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs](https://mlsanity.com/paper/2606.30810/) (2026)
- [Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition](https://mlsanity.com/paper/2606.03657/) (2026)
- [Mind the Gaps: From Failure Attribution to Closed-Form Repair of Code Language Models](https://mlsanity.com/paper/2610.05277/) (2026)

## Abstract (arXiv)

Large Language Models for Code (LLMs4Code) have been found to exhibit outstanding performance in the software engineering domain, especially the remarkable performance in coding tasks. However, even the most advanced LLMs4Code can inevitably contain incorrect or outdated code knowledge. Due to the high cost of training LLMs4Code, it is impractical to re-train the models for fixing these problematic code knowledge. Model editing is a new technical field for effectively and efficiently correcting erroneous knowledge in LLMs, where various model editing techniques and benchmarks have been proposed recently. Despite that, a comprehensive study that thoroughly compares and analyzes the performance of the state-of-the-art model editing techniques for adapting the knowledge within LLMs4Code across various code-related tasks is notably absent. To bridge this gap, we perform the first systematic study on applying state-of-the-art model editing approaches to repair the inaccuracy of LLMs4Code. To that end, we introduce a benchmark named CLMEEval, which consists of two datasets, i.e., CoNaLa-Edit (CNLE) with 21K+ code generation samples and CodeSearchNet-Edit (CSNE) with 16K+ code summarization samples. With the help of CLMEEval, we evaluate six advanced model editing techniques on three LLMs4Code: CodeLlama (7B), CodeQwen1.5 (7B), and Stable-Code (3B). Our findings include that the external memorization-based GRACE approach achieves the best knowledge editing effectiveness and specificity (the editing does not influence untargeted knowledge), while generalization (whether the editing can generalize to other semantically-identical inputs) is a universal challenge for existing techniques. Furthermore, building on in-depth case analysis, we introduce an enhanced version of GRACE called A-GRACE, which incorporates contrastive learning to better capture the semantics of the inputs.