mlsanityData to 7 Oct 2026

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

arXiv 2102.04664 ↗First posted 9 Feb 2021cs.SE cs.CLcited by 732

Not on the map. Why?

Cited by 53 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

Key facts. Cited by 732 papers in the map (data through 7 Oct 2026). Builds on 56 of its 117 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

56 of 117 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
38 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2018code2vec: Learning Distributed Representations of Codecited by 249
  10. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  11. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  12. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  13. 2017Learning to Represent Programs with Graphscited by 219
  14. 2019SPoC: Search-based Pseudocode to Codecited by 176
  15. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  16. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  17. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  18. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  19. 2018Mapping Language to Code in Programmatic Contextcited by 104
  20. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
  21. 2015Neural Programmer-Interpreterscited by 96
  22. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  23. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  24. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  25. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  26. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  27. 2019Code Generation as a Dual Task of Code Summarizationcited by 62
  28. 2018Neural Code Comprehension: A Learnable Representation of Code Semanticscited by 59
  29. 2019Pythia: AI-assisted Code Completion Systemcited by 54
  30. 2018Structured Neural Summarizationcited by 53
  31. 2020Big Code != Big Vocabulary: Open-Vocabulary Models for Source Codecited by 53
  32. 2019Neural Program Repair by Jointly Learning to Localize and Repaircited by 39
  33. 2020Incorporating External Knowledge through Pre-training for Natural Language to Code Generationcited by 36
  34. 2015On the "Naturalness" of Buggy Codecited by 35
  35. 2015Neural Programmer: Inducing Latent Programs with Gradient Descentcited by 32
  36. 2018StaQC: A Systematically Mined Question-Code Dataset from Stack Overflowcited by 30
  37. 2018Aroma: Code Recommendation via Structural Code Searchcited by 29
  38. 2014Mining Idioms from Source Codecited by 19

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 99; 2023: 168; 2024: 184; 2025: 156; 2026: 80.

732 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  10. 2024A Survey on Large Language Models for Code Generationcited by 646
  11. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  12. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
38 more
  1. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  3. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  4. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  6. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  10. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  12. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  13. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  14. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  15. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  16. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  17. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  18. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  19. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  20. 2023SantaCoder: don't reach for the stars!cited by 169
  21. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  22. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  23. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  24. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  25. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  26. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  27. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  28. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  29. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
  30. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  31. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  32. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  33. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  34. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  35. 2022Natural Language to Code Translation with Executioncited by 122
  36. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  37. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  38. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110

50 most cited of 732 All 732 (JSON)