Self-Explained Keywords Empower Large Language Models for Code Generation
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have achieved impressive performance in code generation. However, due to the long-tail distribution of LLMs' training data, low-frequency terms are typically underrepresented in the training process. Consequently, LLMs often misunderstand or overlook problem-specific, low-frequency keywords during code generation, compromising the accuracy of the generated code. To address this, we propose a novel technique named SEK(Self-Explained Keywords), which empowers an LLM for better code generation by extracting and explaining the key terms in the problem description with the LLM itself and ranking them based on frequency. Comprehensive experiments across three benchmarks, i.e., HumanEval(+), MBPP(+), and APPS, with five representative LLMs, show that SEK can significantly improve LLMs in code generation, yielding substantial and consistent gains. For instance, SEK improves the Pass@1 of DeepSeek-Coder-V2-Instruct from 85.4% to 93.3% on the Humaneval benchmark. Further analysis confirms that SEK enables the LLMs to shift their attention from low-frequency keywords to their corresponding high-frequency counterparts.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 51 of its 93 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
51 of 93 references.
References · 51
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2023Teaching Large Language Models to Self-Debugcited by 988
- 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
38 more
- 2021Measuring Coding Challenge Competence With APPScited by 902
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
- 2023StarCoder: may the source be with you!cited by 799
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2022Memory-Based Model Editing at Scalecited by 447
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
- 2022CodeT: Code Generation with Generated Testscited by 384
- 2021Unified Pre-training for Program Understanding and Generationcited by 366
- 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
- 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
- 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
- 2023Self-collaboration Code Generation via ChatGPTcited by 258
- 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
- 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
- 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
- 2023Self-planning Code Generation with Large Language Modelscited by 124
- 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
- 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
- 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
- 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
- 2018Simple Unsupervised Keyphrase Extraction using Sentence Embeddingscited by 62
- 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
- 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
- 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 42
- 2024LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extractioncited by 39
- 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 38
- 2016Mutual Information and Diverse Decoding Improve Neural Machine Translationcited by 35
- 2022Fault-Aware Neural Code Rankerscited by 33
- 2023Zero-Shot Detection of Machine-Generated Codescited by 24
- 2023Fixing Large Language Models' Specification Misunderstanding for Better Code Generationcited by 17
- 2024Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understandingcited by 9
- 2024B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Testscited by 9
- 2023LLM-TAKE: Theme Aware Keyword Extraction Using Large Language Modelscited by 8
- 2024JumpCoder: Go Beyond Autoregressive Coder via Online Modificationcited by 7
- 2024Knowledge-Aware Code Generation with Large Language Modelscited by 6
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.