mlsanityData to 7 Oct 2026

CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation

arXiv 2206.06888 ↗First posted 14 Jun 2022cs.SE cs.CL cs.PLcited by 87

Not on the map. Why?

Abstract (arXiv)

Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large-scale unlabelled code corpora and perform well in code generation. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch. Both the sketcher and the generator are continually pre-trained upon a base model using unlabelled data. Furthermore, we craft two benchmarks named PandasEval and NumpyEval to evaluate library-oriented code generation. Experimental results demonstrate the impressive performance of CERT. For example, it surpasses the base model by an absolute 15.67% improvement in terms of pass@1 on PandasEval. Our work is available at https://github.com/microsoft/PyCodeGPT.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

17 of 28 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
5 more
  1. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  3. 2018Coarse-to-Fine Decoding for Neural Semantic Parsingcited by 110
  4. 2019Learning to Infer Program Sketchescited by 44
  5. 2018A Grammar-Based Structural CNN Decoder for Code Generationcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 42; 2025: 20; 2026: 5.

87 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  5. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  6. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  7. 2024Continual Learning for Large Language Models: A Surveycited by 144
  8. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  9. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  10. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  11. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  12. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
38 more
  1. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  2. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  3. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  4. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  5. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  6. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  7. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  8. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  9. 2022When Language Model Meets Private Librarycited by 54
  10. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  11. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  12. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  13. 2023Data Contamination Through the Lens of Timecited by 39
  14. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  15. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  16. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  17. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 33
  18. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  19. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisationcited by 31
  20. 2024Benchmarking Data Science Agentscited by 30
  21. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  22. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  23. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  24. 2023Traces of Memorisation in Large Language Models for Codecited by 26
  25. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  26. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  27. 2023Zero-Shot Detection of Machine-Generated Codescited by 24
  28. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
  29. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22
  30. 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
  31. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  32. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 17
  33. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  34. 2023Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigationcited by 14
  35. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 13
  36. 2024Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmarkcited by 13
  37. 2024JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Modelscited by 10
  38. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10

50 most cited of 87 All 87 (JSON)