mlsanityData to 7 Oct 2026

Better Language Models of Code through Self-Improvement

arXiv 2304.01228 ↗First posted 2 Apr 2023cs.CL cs.AIcited by 14

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Pre-trained language models for code (PLMCs) have gained attention in recent research. These models are pre-trained on large-scale datasets using multi-modal objectives. However, fine-tuning them requires extensive supervision and is limited by the size of the dataset provided. We aim to improve this issue by proposing a simple data augmentation framework. Our framework utilizes knowledge gained during the pre-training and fine-tuning stage to generate pseudo data, which is then used as training data for the next step. We incorporate this framework into the state-of-the-art language models, such as CodeT5, CodeBERT, and UnixCoder. The results show that our framework significantly improves PLMCs' performance in code-related sequence generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 22 of its 31 references that are in the map. First posted on arXiv 2 Apr 2023.

Builds on

22 of 31 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
10 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  2. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  3. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  4. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  5. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  6. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  7. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  8. 2021Multilingual training for Software Engineeringcited by 43
  9. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  10. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 7.

14 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 23
  3. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  4. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
  5. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  6. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  7. 2023Functional Overlap Reranking for Neural Code Generationcited by 9
  8. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  9. 2023Synthetic pre-training for neural-network interatomic potentialscited by 6
  10. 2024XMainframe: A Large Language Model for Mainframe Modernizationcited by 6
  11. 2024VisualCoder: Guiding Large Language Models in Code Execution with Fine-grained Multimodal Chain-of-Thought Reasoningcited by 3
  12. 2023Learning Curricula in Open-Ended Worldscited by 2
2 more
  1. 2024Envisioning the Next-Generation AI Coding Assistants: Insights & Proposalscited by 2
  2. 2024Dopamin: Transformer-based Comment Classifiers through Domain Post-Training and Multi-level Layer Aggregationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
  2. 2024Improving the Ability of Pre-trained Language Model by Imparting Large Language Model's Experience ↗
  3. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  4. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  5. 2024Assessing Code Generation with Intermediate Languages
  6. 2025A Code Comprehension Benchmark for Large Language Models for Code
  7. 2024Case2Code: Scalable Synthetic Data for Code Generation
  8. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation