# CodeQueries: A Dataset of Semantic Queries over Code

arXiv: https://arxiv.org/abs/2209.08372 · HTML: https://mlsanity.com/paper/2209.08372/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019)
- [LambdaNet: Probabilistic Type Inference using Graph Neural Networks](https://mlsanity.com/paper/2005.02161/) (2020)
- [Making Neural QA as Simple as Possible but not Simpler](https://mlsanity.com/paper/1703.04816/) (2017)
- [TypeWriter: Neural Type Prediction with Search-based Validation](https://mlsanity.com/paper/1912.03768/) (2019)
- [StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow](https://mlsanity.com/paper/1803.09371/) (2018)
- [CodeQA: A Question Answering Dataset for Source Code Comprehension](https://mlsanity.com/paper/2109.08365/) (2021)
- [Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent](https://mlsanity.com/paper/2008.12193/) (2020)
- [OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints](https://mlsanity.com/paper/2004.00348/) (2020)
- [CRaDLe: Deep Code Retrieval Based on Semantic Dependency Learning](https://mlsanity.com/paper/2012.01028/) (2020)
- [Static Inference Meets Deep Learning: A Hybrid Type Inference Approach for Python](https://mlsanity.com/paper/2105.03595/) (2021)
- [Learning a Static Analyzer from Data](https://mlsanity.com/paper/1611.01752/) (2016)
- [A Neural Question Answering System for Basic Questions about Subroutines](https://mlsanity.com/paper/2101.03999/) (2021)
- [Active Learning of Points-To Specifications](https://mlsanity.com/paper/1711.03239/) (2017)

## Built on by (most cited first, 11 of 11)

- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding](https://mlsanity.com/paper/2603.16124/) (2026)
- [Leveraging Large Language Models in Code Question Answering: Baselines and Issues](https://mlsanity.com/paper/2411.03012/) (2024)
- [Frustrated with Code Quality Issues? LLMs can Help!](https://mlsanity.com/paper/2309.12938/) (2023)
- [Gistify! Codebase-Level Understanding via Runtime Execution](https://mlsanity.com/paper/2510.26790/) (2025)
- [RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deployment](https://mlsanity.com/paper/2509.10436/) (2025)
- [LensVLM: Selective Context Expansion for Compressed Visual Representation of Text](https://mlsanity.com/paper/2605.07019/) (2026)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration](https://mlsanity.com/paper/2608.24221/) (2026)
- [Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark](https://mlsanity.com/paper/2609.28449/) (2026)

## Abstract (arXiv)

Developers often have questions about semantic aspects of code they are working on, e.g., "Is there a class whose parent classes declare a conflicting attribute?". Answering them requires understanding code semantics such as attributes and inheritance relation of classes. An answer to such a question should identify code spans constituting the answer (e.g., the declaration of the subclass) as well as supporting facts (e.g., the definitions of the conflicting attributes). The existing work on question-answering over code has considered yes/no questions or method-level context. We contribute a labeled dataset, called CodeQueries, of semantic queries over Python code. Compared to the existing datasets, in CodeQueries, the queries are about code semantics, the context is file level and the answers are code spans. We curate the dataset based on queries supported by a widely-used static analysis tool, CodeQL, and include both positive and negative examples, and queries requiring single-hop and multi-hop reasoning. To assess the value of our dataset, we evaluate baseline neural approaches. We study a large language model (GPT3.5-Turbo) in zero-shot and few-shot settings on a subset of CodeQueries. We also evaluate a BERT style model (CuBERT) with fine-tuning. We find that these models achieve limited success on CodeQueries. CodeQueries is thus a challenging dataset to test the ability of neural models, to understand code semantics, in the extractive question-answering setting.