# CoSQA: 20,000+ Web Queries for Code Search and Question Answering

arXiv: https://arxiv.org/abs/2105.13239 · HTML: https://mlsanity.com/paper/2105.13239/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 25 of 25)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [A parallel corpus of Python functions and documentation strings for automated code documentation and code generation](https://mlsanity.com/paper/1707.02275/) (2017)
- [Contrastive Distillation on Intermediate Representations for Language Model Compression](https://mlsanity.com/paper/2009.14167/) (2020)
- [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019)
- [Natural Language Inference by Tree-Based Convolution and Heuristic Matching](https://mlsanity.com/paper/1512.08422/) (2015)
- [Learning Semantic Textual Similarity from Conversations](https://mlsanity.com/paper/1804.07754/) (2018)
- [Multi-Modal Attention Network Learning for Semantic Source Code Retrieval](https://mlsanity.com/paper/1909.13516/) (2019)
- [StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow](https://mlsanity.com/paper/1803.09371/) (2018)
- [CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning](https://mlsanity.com/paper/1904.00720/) (2019)
- [Opportunities and Challenges in Code Search Tools](https://mlsanity.com/paper/2011.02297/) (2020)
- [A Multi-Perspective Architecture for Semantic Code Search](https://mlsanity.com/paper/2005.06980/) (2020)
- [Query Expansion Based on Crowd Knowledge for Code Search](https://mlsanity.com/paper/1703.01443/) (2017)
- [Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent](https://mlsanity.com/paper/2008.12193/) (2020)
- [Neural Code Search Evaluation Dataset](https://mlsanity.com/paper/1908.09804/) (2019)
- [Adversarial Training for Code Retrieval with Question-Description Relevance Regularization](https://mlsanity.com/paper/2010.09803/) (2020)

## Built on by (most cited first, 30 of 75)

- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [At Which Training Stage Does Code Data Help LLMs Reasoning?](https://mlsanity.com/paper/2309.16298/) (2023)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking](https://mlsanity.com/paper/2412.01007/) (2024)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [CoCoSoDa: Effective Contrastive Learning for Code Search](https://mlsanity.com/paper/2204.03293/) (2022)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search](https://mlsanity.com/paper/2201.10866/) (2022)
- [SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner](https://mlsanity.com/paper/2506.09003/) (2025)
- [ContraCLM: Contrastive Learning For Causal Language Model](https://mlsanity.com/paper/2210.01185/) (2022)
- [Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search](https://mlsanity.com/paper/2401.04514/) (2024)
- [CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training](https://mlsanity.com/paper/2205.02029/) (2022)
- [O1 Embedder: Let Retrievers Think Before Action](https://mlsanity.com/paper/2502.07555/) (2025)
- [ProBench: Benchmarking Large Language Models in Competitive Programming](https://mlsanity.com/paper/2502.20868/) (2025)
- [Rethinking Negative Pairs in Code Search](https://mlsanity.com/paper/2310.08069/) (2023)
- [ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks](https://mlsanity.com/paper/2305.16837/) (2023)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [CodeQueries: A Dataset of Semantic Queries over Code](https://mlsanity.com/paper/2209.08372/) (2022)
- [CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents](https://mlsanity.com/paper/2406.11589/) (2024)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models](https://mlsanity.com/paper/2406.01359/) (2024)
- [SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding](https://mlsanity.com/paper/2603.16124/) (2026)
- [Exploring Representation-Level Augmentation for Code Search](https://mlsanity.com/paper/2210.12285/) (2022)
- [Code Representation Pre-training with Complements from Program Executions](https://mlsanity.com/paper/2309.09980/) (2023)
- [Code Membership Inference for Detecting Unauthorized Data Use in Code Pre-trained Language Models](https://mlsanity.com/paper/2312.07200/) (2023)

## Abstract (arXiv)

Finding codes given natural language query isb eneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce the CoSQA dataset.It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance query-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%, and incorporating CoCLR brings a further improvement of 10.5%.