A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generation
Not on the map. Why?
Abstract (arXiv)
Language Models (LMs) are widely used in software engineering for code generation, but they may produce erroneous code. Rather than repairing outputs, a more thorough remedy is to address underlying model failures. LM repair offers a lightweight solution: it requires minimal data, lowers computational cost, and limits side effects. Unlike full retraining, LM repair focuses on applying tailored updates to targeted neurons, making it suitable for limited resources, high-performance demands, or strict safety requirements. In this paper, we propose Semantic Targeting for Analytical Repair (STAR), a novel semantic-based optimization method for repairing LLMs. STAR realizes the main operations of repairing LMs in an optimization process, including locating ``buggy neurons'', solving ``neuron patches'', and patching ``buggy neurons''. The neuron patches are computed with a solid semantic-based analytical formula, which directly bridges the changes to logits with the deltas of neurons, by steering latent representations. Compared to the prior work of LM repair (MINT) and standard optimization methods (SGD), STAR integrates their strengths while mitigating their limitations. By reformulating LM repair as an optimization process, STAR may solve multiple failures together, significantly improving the usefulness. Evaluated on coding tasks using popular code LMs, STAR demonstrates superior effectiveness compared with the state-of-the-art. Besides, STAR exhibits better efficiency. In terms of side effects, namely the balance between generalization and specificity, STAR outperforms prior work by a significant margin. Additionally, we conducted assessments on the overfitting risk of LM repair as well as the cumulative impact. Further, we analyzed the differences with pipeline-based methods and explained the reason why STAR is better and how it mitigated the common limitations of LM repair.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 17 Mar 2025.
Builds on
44 of 67 references.
References · 44
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2017Mixed Precision Trainingcited by 1,135
- 2022Mass-Editing Memory in a Transformercited by 976
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2024A Survey on Large Language Models for Code Generationcited by 646
- 2021Knowledge Neurons in Pretrained Transformerscited by 590
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
32 more
- 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
- 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
- 2018Techniques for Interpretable Machine Learningcited by 237
- 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
- 2023PMET: Precise Model Editing in a Transformercited by 170
- 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
- 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
- 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
- 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
- 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
- 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
- 2024BadEdit: Backdooring large language models by model editingcited by 84
- 2019Neural Network Attributions: A Causal Perspectivecited by 77
- 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
- 2017Towards better understanding of gradient-based attribution methods for Deep Neural Networkscited by 51
- 2022Causality-based Neural Network Repaircited by 44
- 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
- 2023Code Execution with Pre-trained Language Modelscited by 33
- 2019Arachne: Search Based Repair of Deep Neural Networkscited by 25
- 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
- 2022FairNeuron: Improving Deep Neural Network Fairness with Adversary Games on Selective Neuronscited by 20
- 2021Shellcode_IA32: A Dataset for Automatic Shellcode Generationcited by 16
- 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoningcited by 15
- 2024Identifying Query-Relevant Neurons in Large Language Models for Long-Form Textscited by 12
- 2023Understanding and Unifying Fourteen Attribution Methods with Taylor Interactionscited by 11
- 2024From Today's Code to Tomorrow's Symphony: The AI Transformation of Developer's Routine by 2030cited by 8
- 2024Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Modelscited by 8
- 2024Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learningcited by 7
- 2024A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Modelscited by 5
- 2023Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generationcited by 4
- 2022Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executablescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.