mlsanityData to 7 Oct 2026

GraphCodeBERT: Pre-training Code Representations with Data Flow

arXiv 2009.08366 ↗First posted 17 Sep 2020cs.SE cs.CLcited by 600

Not on the map. Why?

Cited by 34 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Key facts. Cited by 600 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 93; 2023: 136; 2024: 108; 2025: 125; 2026: 102.

600 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2022Multimodal Learning with Transformers: A Surveycited by 399
  7. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  8. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  9. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  10. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  11. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
38 more
  1. 2023Planning with Large Language Models for Code Generationcited by 179
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  3. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  4. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  5. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  6. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  7. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  8. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  9. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  10. 2023ChipGPT: How far are we from natural language hardware designcited by 101
  11. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  12. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  13. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  14. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  15. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  16. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  17. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  18. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  19. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  20. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  21. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  22. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  23. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  24. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  25. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  26. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  27. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  28. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  29. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  30. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  31. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  32. 2022When Language Model Meets Private Librarycited by 54
  33. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  34. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  35. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  36. 2023Large Language Models for Compiler Optimizationcited by 52
  37. 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
  38. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50

50 most cited of 600 All 600 (JSON)