mlsanityData to 14 Oct 2026

Assessing the Answerability of Queries in Retrieval-Augmented Code Generation

arXiv 2411.05547 ↗First posted 8 Nov 2024cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG for code

Abstract (arXiv)

Thanks to unprecedented language understanding and generation capabilities of large language model (LLM), Retrieval-augmented Code Generation (RaCG) has recently been widely utilized among software developers. While this has increased productivity, there are still frequent instances of incorrect codes being provided. In particular, there are cases where plausible yet incorrect codes are generated for queries from users that cannot be answered with the given queries and API descriptions. This study proposes a task for evaluating answerability, which assesses whether valid answers can be generated based on users' queries and retrieved APIs in RaCG. Additionally, we build a benchmark dataset called Retrieval-augmented Code Generability Evaluation (RaCGEval) to evaluate the performance of models performing this task. Experimental results show that this task remains at a very challenging level, with baseline models exhibiting a low performance of 46.7%. Furthermore, this study discusses methods that could significantly improve performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 8 of its 17 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

8 of 17 references.

References · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  5. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  6. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  7. 2022When Language Model Meets Private Librarycited by 54
  8. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
  2. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
  3. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?
  4. 2024EVOR: Evolving Retrieval for Code Generation
  5. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
  6. 2026Context-Augmented Code Generation Using Programming Knowledge Graphs
  7. 2026RCL: A Retrieval-Confidence Layer for Detecting Insufficient Context in Enterprise Retrieval-Augmented Code Generation
  8. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation