# CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation arXiv: https://arxiv.org/abs/2405.02355 ยท HTML: https://mlsanity.com/paper/2405.02355/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. - Builds on 26 of its 40 references that are in the map. - First posted on arXiv 3 May 2024. ## Placed in - Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/) ## Builds on (most cited first, 26 of 26) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021) - [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018) - [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017) - [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023) - [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022) - [PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback](https://mlsanity.com/paper/2307.14936/) (2023) - [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018) - [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019) - [TreeBERT: A Tree-Based Pre-Trained Model for Programming Language](https://mlsanity.com/paper/2105.12485/) (2021) - [Automated software vulnerability detection with machine learning](https://mlsanity.com/paper/1803.04497/) (2018) - [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021) - [EVOR: Evolving Retrieval for Code Generation](https://mlsanity.com/paper/2402.12317/) (2024) - [Multimodal Representation for Neural Code Search](https://mlsanity.com/paper/2107.00992/) (2021) - [Multi-View Graph Representation for Programming Language Processing: An Investigation into Algorithm Detection](https://mlsanity.com/paper/2202.12481/) (2022) ## Built on by (most cited first, 4 of 4) - [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025) - [Selective Prompt Anchoring for Code Generation](https://mlsanity.com/paper/2408.09121/) (2024) - [HDLxGraph: Bridging Large Language Models and HDL Repositories via HDL Graph Databases](https://mlsanity.com/paper/2505.15701/) (2025) - [A Methodology for Investigating AI Patterns Prevalence in Software Repositories](https://mlsanity.com/paper/2607.00558/) (2026) ## Abstract (arXiv) Utilizing large language models to generate codes has shown promising meaning in software development revolution. Despite the intelligence shown by the large language models, their specificity in code generation can still be improved due to the syntactic gap and mismatched vocabulary existing between natural language and programming languages. In this paper, we propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework that bridges the gap between NL and PL to enhance the performance of LLMs. CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to better interpret the programming domain knowledge, which can facilitate natural language based LLMs for better understanding of code syntax and serve as a bridge among different programming languages. To take the extracted structural knowledge into the foundation models, we propose 1) a hard meta-graph prompt template to transform the challenging syntax graph into informative graphical view for tuning-free models and 2) a soft prompting technique that injects the domain knowledge of programming languages into model parameters via finetuning the models with the soft signals encoded by GNN expert model. Specifically, two constraints are designed to improve the alignment and structure expressiveness, contributing to the informativeness of the single-token-sized external for enhanced code generation. CodeGRAG significantly improves the code generation ability of LLMs and can even offer performance gain for cross-lingual code generation. Implementation is available at https://anonymous.4open.science/r/Code-5970/ .