mlsanityData to 7 Oct 2026

CoSQA: 20,000+ Web Queries for Code Search and Question Answering

arXiv 2105.13239 ↗First posted 27 May 2021cs.CL cs.SEcited by 74

Not on the map. Why?

Abstract (arXiv)

Finding codes given natural language query isb eneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce the CoSQA dataset.It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance query-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%, and incorporating CoCLR brings a further improvement of 10.5%.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 27 May 2021.

Builds on

25 of 42 references.

References · 25

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  5. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  6. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  9. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  10. 2016Enhanced LSTM for Natural Language Inferencecited by 199
  11. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  12. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
13 more
  1. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 56
  2. 2019When Deep Learning Met Code Searchcited by 45
  3. 2015Natural Language Inference by Tree-Based Convolution and Heuristic Matchingcited by 42
  4. 2018Learning Semantic Textual Similarity from Conversationscited by 41
  5. 2019Multi-Modal Attention Network Learning for Semantic Source Code Retrievalcited by 40
  6. 2018StaQC: A Systematically Mined Question-Code Dataset from Stack Overflowcited by 30
  7. 2019CoaCor: Code Annotation for Code Retrieval with Reinforcement Learningcited by 22
  8. 2020Opportunities and Challenges in Code Search Toolscited by 22
  9. 2020A Multi-Perspective Architecture for Semantic Code Searchcited by 20
  10. 2017Query Expansion Based on Crowd Knowledge for Code Searchcited by 18
  11. 2020Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intentcited by 16
  12. 2019Neural Code Search Evaluation Datasetcited by 13
  13. 2020Adversarial Training for Code Retrieval with Question-Description Relevance Regularizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 14; 2024: 14; 2025: 22; 2026: 10.

74 papers.

Sort:

By area:

  1. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  2. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  5. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  6. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  7. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?cited by 37
  8. 2024Code Representation Learning At Scalecited by 26
  9. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  10. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  11. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  12. 2022CoCoSoDa: Effective Contrastive Learning for Code Searchcited by 21
38 more
  1. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  2. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 18
  3. 2024Rewriting the Code: A Simple Method for Large Language Model Augmented Code Searchcited by 17
  4. 2022CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Trainingcited by 15
  5. 2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
  6. 2025ProBench: Benchmarking Large Language Models in Competitive Programmingcited by 15
  7. 2023Rethinking Negative Pairs in Code Searchcited by 14
  8. 2023ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Taskscited by 12
  9. 2024ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Searchcited by 12
  10. 2022CodeQueries: A Dataset of Semantic Queries over Codecited by 11
  11. 2024CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agentscited by 11
  12. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  13. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 10
  14. 2024R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Modelscited by 9
  15. 2026SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understandingcited by 9
  16. 2022Exploring Representation-Level Augmentation for Code Searchcited by 8
  17. 2023Code Representation Pre-training with Complements from Program Executionscited by 8
  18. 2022Soft-Labeled Contrastive Pre-training for Function-level Code Representationcited by 7
  19. 2026CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Codingcited by 7
  20. 2023GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understandingcited by 6
  21. 2024Mitigating Data Imbalance for Software Vulnerability Assessment: Does Data Augmentation Help?cited by 6
  22. 2025CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrievalcited by 6
  23. 2025MIRB: Mathematical Information Retrieval Benchmarkcited by 5
  24. 2021Cascaded Fast and Slow Models for Efficient Semantic Code Searchcited by 4
  25. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  26. 2023Code Search Debiasing:Improve Search Results beyond Overall Ranking Performancecited by 4
  27. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarkingcited by 4
  28. 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
  29. 2022NS3: Neuro-Symbolic Semantic Code Searchcited by 3
  30. 2022Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelinescited by 3
  31. 2024Towards Better Code Understanding in Decoder-Only Models with Contrastive Learningcited by 3
  32. 2024Contextualized Data-Wrangling Code Generation in Computational Notebookscited by 3
  33. 2024Code LLMs: A Taxonomy-based Surveycited by 3
  34. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  35. 2025CoRet: Improved Retriever for Code Editingcited by 3
  36. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  37. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  38. 2023Laminar: A New Serverless Stream-based Framework with Semantic Code Search and Code Completioncited by 2

50 most cited of 74 All 74 (JSON)