mlsanityData to 14 Oct 2026

Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation

arXiv 2312.05356 ↗First posted 8 Dec 2023cs.SE cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) have become widely used in software engineering, especially for tasks such as code generation, where they are referred to as code LMs. These models have proven effective in generating code, making it easier for developers to automate coding activities. However, research has highlighted a significant limitation: despite their effectiveness, LMs often produce code that is incorrect, buggy, or not fully functional. Updating these models with limited data can be prohibitively challenging, yet it is essential to maximize their utility. This may require hot-fix techniques (updating models with limited data) to resolve. In this paper, we propose ulModel ulImprovement via ulNeuron ulTargeting (textscMINT), a novel approach for repairing code LMs. MINT leverages the semantic property of language models to perform neuron-level repairs in a novel way. Further, by analyzing the relationships between the model's latent representations, the incorrect outputs, and the desired outputs, textscMINT determines which neurons are worth updating. This approach ensures that only the neurons crucial to the model's failure are targeted, avoiding unnecessary changes and allowing for a more efficient and precise repair process. textscMINT is effective, efficient, and reliable, capable of correcting a neural model by patching a minimum number of neurons (usually one or two neurons). Our approach is evaluated on three coding tasks: line-level code generation, shellcode generation, and intent-to-bash translation. The experimental results demonstrate that the proposed approach significantly outperforms the state-of-the-art in both effectiveness and efficiency measures. In addition, we analyze and discuss the side effects of model repair techniques, including the balance between generalization and specificity, and the performance after multiple repairs in succession.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

50 of 79 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
38 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2022Mass-Editing Memory in a Transformercited by 1,001
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  9. 2021Fast Model Editing at Scalecited by 498
  10. 2022Memory-Based Model Editing at Scalecited by 447
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  12. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  13. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  14. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  15. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 328
  16. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  17. 2023PMET: Precise Model Editing in a Transformercited by 173
  18. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  19. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  20. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  21. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  22. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  23. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  24. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  25. 2018How Important Is a Neuron?cited by 73
  26. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  27. 2017Towards better understanding of gradient-based attribution methods for Deep Neural Networkscited by 54
  28. 2021Towards Automating Code Review Activitiescited by 54
  29. 2022Causality-based Neural Network Repaircited by 47
  30. 2023Code Execution with Pre-trained Language Modelscited by 34
  31. 2022Repairing Bugs in Python Assignments Using Large Language Modelscited by 28
  32. 2019Arachne: Search Based Repair of Deep Neural Networkscited by 26
  33. 2022FairNeuron: Improving Deep Neural Network Fairness with Adversary Games on Selective Neuronscited by 21
  34. 2022Assemble Foundation Models for Automatic Code Summarizationcited by 17
  35. 2021Shellcode_IA32: A Dataset for Automatic Shellcode Generationcited by 16
  36. 2023Understanding and Unifying Fourteen Attribution Methods with Taylor Interactionscited by 11
  37. 2016An Analysis of the Search Spaces for Generate and Validate Patch Generation Systemscited by 9
  38. 2022Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executablescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Modelscited by 6
  2. 2025A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generationcited by 2
  3. 2026Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablationcited by 2
  4. 2025Understanding Robustness of Model Editing in Code LLMscited by 1
  5. 2026Mind the Gaps: From Failure Attribution to Closed-Form Repair of Code Language Modelscited by 0