mlsanityData to 14 Oct 2026

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

arXiv 2002.08155 ↗First posted 19 Feb 2020cs.CL cs.PLcited by 1,310

Not on the map. Why?

Cited by 82 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.

Key facts. Cited by 1,310 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 93; 2022: 174; 2023: 276; 2024: 264; 2025: 301; 2026: 202.

1,310 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2023StarCoder: may the source be with you!cited by 799
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
38 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 518
  3. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  6. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  7. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  8. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  9. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  12. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  13. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  14. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  15. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  16. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  17. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  18. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  19. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  20. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  21. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  22. 2023Planning with Large Language Models for Code Generationcited by 182
  23. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  24. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  25. 2023SantaCoder: don't reach for the stars!cited by 170
  26. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  27. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  28. 2021Retrieval Augmented Code Generation and Summarizationcited by 149
  29. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  30. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  31. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  32. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  33. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  34. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  35. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  36. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  37. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  38. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 119

50 most cited of 1,310 All 1,310 (JSON)