mlsanityData to 7 Oct 2026

Code Needs Comments: Enhancing Code LLMs with Comment Augmentation

arXiv 2402.13013 ↗First posted 20 Feb 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs' performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 19 of its 51 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

19 of 51 references.

References · 19

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
7 more
  1. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  2. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  3. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  4. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  5. 2019Misleading Authorship Attribution of Source Code using Adversarial Learningcited by 33
  6. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  7. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  3. 2024TheoremLlama: Transforming General-Purpose LLMs into Lean4 Expertscited by 58
  4. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  5. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  6. 2025Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferencescited by 10
  7. 2025VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generationcited by 8
  8. 2024AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Datacited by 6
  9. 2025CritiQ: Mining Data Quality Criteria from Human Preferencescited by 6
  10. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  11. 2025AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robotcited by 5
  12. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarkingcited by 4
4 more
  1. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 2
  2. 2026Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentationcited by 1
  3. 2026Learning Generalizable Multimodal Representations for Software Vulnerability Detectioncited by 0
  4. 2026CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernizationcited by 0