# CodeQA: A Question Answering Dataset for Source Code Comprehension

arXiv: https://arxiv.org/abs/2109.08365 · HTML: https://mlsanity.com/paper/2109.08365/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering](https://mlsanity.com/paper/2011.03080/) (2020)
- [A parallel corpus of Python functions and documentation strings for automated code documentation and code generation](https://mlsanity.com/paper/1707.02275/) (2017)
- [Automatic Source Code Summarization with Extended Tree-LSTM](https://mlsanity.com/paper/1906.08094/) (2019)

## Built on by (most cited first, 30 of 30)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [Retraining-Free Merging of Sparse MoE via Hierarchical Clustering](https://mlsanity.com/paper/2410.08589/) (2024)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [UniCoder: Scaling Code Large Language Model via Universal Code](https://mlsanity.com/paper/2406.16441/) (2024)
- [CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs](https://mlsanity.com/paper/2410.01999/) (2024)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [CodeQueries: A Dataset of Semantic Queries over Code](https://mlsanity.com/paper/2209.08372/) (2022)
- [CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course](https://mlsanity.com/paper/2210.14494/) (2022)
- [CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering](https://mlsanity.com/paper/2412.14764/) (2024)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding](https://mlsanity.com/paper/2603.16124/) (2026)
- [Leveraging Large Language Models in Code Question Answering: Baselines and Issues](https://mlsanity.com/paper/2411.03012/) (2024)
- [KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?](https://mlsanity.com/paper/2601.13240/) (2026)
- [NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding](https://mlsanity.com/paper/2504.09249/) (2025)
- [Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executables](https://mlsanity.com/paper/2209.01566/) (2022)
- [Semantic-Preserving Adversarial Code Comprehension](https://mlsanity.com/paper/2209.05130/) (2022)
- [SWE-QA: A Dataset and Benchmark for Complex Code Understanding](https://mlsanity.com/paper/2604.24814/) (2026)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [Empowering RepoQA-Agent based on Reinforcement Learning Driven by Monte-carlo Tree Search](https://mlsanity.com/paper/2510.26287/) (2025)
- [Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding](https://mlsanity.com/paper/2601.15339/) (2026)
- [Neurosymbolic Repo-level Code Localization](https://mlsanity.com/paper/2604.16021/) (2026)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning](https://mlsanity.com/paper/2601.21649/) (2026)
- [DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration](https://mlsanity.com/paper/2608.24221/) (2026)
- [Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark](https://mlsanity.com/paper/2609.28449/) (2026)

## Abstract (arXiv)

We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.