mlsanityData to 7 Oct 2026

PanGu-Coder: Program Synthesis with Function-Level Language Modeling

arXiv 2207.11280 ↗First posted 22 Jul 2022cs.LG cs.AI cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PanGu-Coder using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PanGu-Coder-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PanGu-Coder with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX, while attending a smaller context window and training on less data.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 22 Jul 2022.

Builds on

49 of 72 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
37 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  13. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  14. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  15. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  16. 2020Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networkscited by 563
  17. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  18. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  19. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  20. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  21. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  22. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  23. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  24. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  25. 2020Unsupervised Translation of Programming Languagescited by 262
  26. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  27. 2018code2vec: Learning Distributed Representations of Codecited by 249
  28. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  29. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  30. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  31. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  32. 2020Contrastive Code Representation Learningcited by 76
  33. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  34. 2019Selfie: Self-supervised Pretraining for Image Embeddingcited by 46
  35. 2021TreeBERT: A Tree-Based Pre-Trained Model for Programming Languagecited by 40
  36. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  37. 2022CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Trainingcited by 15

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 22; 2024: 17; 2025: 5.

47 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  3. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  4. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  5. 2023SantaCoder: don't reach for the stars!cited by 169
  6. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  7. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  8. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  9. 2024Large Language Models Meet NLP: A Surveycited by 102
  10. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  11. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  12. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
35 more
  1. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  2. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  3. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  4. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  5. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  6. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  7. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  8. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  9. 2024A Performance Study of LLM-Generated Code on Leetcodecited by 48
  10. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  11. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  12. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  13. 2024REPOFUSE: Repository-Level Code Completion with Fused Dual Contextcited by 32
  14. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  15. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 27
  16. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  17. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  18. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  19. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
  20. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunitiescited by 17
  21. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  22. 2024Code Needs Comments: Enhancing Code LLMs with Comment Augmentationcited by 16
  23. 2023CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Modelscited by 15
  24. 2023Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decodingcited by 14
  25. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  26. 2023Exploring Data Augmentation for Code Generation Taskscited by 7
  27. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7
  28. 2023Type Prediction With Program Decomposition and Fill-in-the-Type Trainingcited by 6
  29. 2023Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesiscited by 4
  30. 2022A Survey on Natural Language Processing for Programmingcited by 3
  31. 2024RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romaniancited by 3
  32. 2024Slice-100K: A Multimodal Dataset for Extrusion-based 3D Printingcited by 3
  33. 2023OpenAi's GPT4 as coding assistantcited by 1
  34. 2024Text-to-Code Generation with Modality-relative Pre-trainingcited by 1
  35. 2024CodingTeachLLM: Empowering LLM's Coding Ability via AST Prior Knowledgecited by 0