mlsanityData to 7 Oct 2026

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

arXiv 1909.09436 ↗First posted 20 Sep 2019cs.LG cs.IR cs.SEcited by 638

Not on the map. Why?

Cited by 28 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

Key facts. Cited by 638 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 103; 2023: 138; 2024: 120; 2025: 128; 2026: 95.

638 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  5. 2023StarCoder: may the source be with you!cited by 790
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  7. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
38 more
  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  3. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  4. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  5. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  7. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  11. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  12. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  13. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  14. 2024Token-Budget-Aware LLM Reasoningcited by 242
  15. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  16. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  17. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  18. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  19. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  20. 2023SantaCoder: don't reach for the stars!cited by 169
  21. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  22. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  23. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  24. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  25. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  26. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  27. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  28. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  29. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  30. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  31. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  32. 2023Online Speculative Decodingcited by 109
  33. 2022Task-aware Retrieval with Instructionscited by 107
  34. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  35. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  36. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  37. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  38. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80

50 most cited of 638 All 638 (JSON)