Neural Code Search Evaluation Dataset
Not on the map. Why?
Abstract (arXiv)
There has been an increase of interest in code search using natural language. Assessing the performance of such code search models can be difficult without a readily available evaluation suite. In this paper, we present an evaluation dataset consisting of natural language query and code snippet pairs, with the hope that future work in this area can use this dataset as a common benchmark. We also provide the results of two code search models ([1] and [6]) from recent work. The evaluation dataset is available at https://github.com/facebookresearch/Neural-Code-Search-Evaluation-Dataset
Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Aug 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 5; 2023: 3; 2024: 0; 2025: 1.
13 papers.
Sort:
- 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
- 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
- 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
- 2022On the Importance of Building High-quality Training Datasets for Neural Code Searchcited by 30
- 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
- 2021GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Searchcited by 15
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
- 2021Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understandingcited by 5
- 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
- 2021Senatus -- A Fast and Accurate Code-to-Code Recommendation Enginecited by 3
- 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
- 2022AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Treescited by 1
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents17 Apr 2025
- 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code14 Nov 2023
- 2023OctoPack: Instruction Tuning Code Large Language Models14 Aug 2023
- 2023Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets19 May 2023
- 2022A Survey of Deep Learning Models for Structural Code Understanding3 May 2022
- 2022On the Importance of Building High-quality Training Datasets for Neural Code Search14 Feb 2022
- 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective10 Feb 2022
- 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search26 Jan 2022
- 2022AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees20 Jan 2022
- 2021Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine5 Nov 2021
- 2021GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Search4 Nov 2021
- 2021Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understanding15 Sep 2021
1 more
- 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
- 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
- 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
- 2022On the Importance of Building High-quality Training Datasets for Neural Code Searchcited by 30
- 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
- 2021GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Searchcited by 15
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
- 2021Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understandingcited by 5
- 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
- 2021Senatus -- A Fast and Accurate Code-to-Code Recommendation Enginecited by 3
- 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
- 2022AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Treescited by 1
12 most cited of 13 All 13 (JSON)
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents17 Apr 2025
- 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code14 Nov 2023
- 2023OctoPack: Instruction Tuning Code Large Language Models14 Aug 2023
- 2023Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets19 May 2023
- 2022A Survey of Deep Learning Models for Structural Code Understanding3 May 2022
- 2022On the Importance of Building High-quality Training Datasets for Neural Code Search14 Feb 2022
- 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective10 Feb 2022
- 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search26 Jan 2022
- 2022AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees20 Jan 2022
- 2021Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine5 Nov 2021
- 2021GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Search4 Nov 2021
- 2021Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understanding15 Sep 2021
12 newest of 13 All 13 (JSON)