# Large Language Model-Aware In-Context Learning for Code Generation

arXiv: https://arxiv.org/abs/2310.09748 · HTML: https://mlsanity.com/paper/2310.09748/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 82 references that are in the map.
- First posted on arXiv 15 Oct 2023.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)

## Built on by (most cited first, 30 of 43)

- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning](https://mlsanity.com/paper/2406.07081/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [An LLM Agent for Automatic Geospatial Data Analysis](https://mlsanity.com/paper/2410.18792/) (2024)
- [From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility](https://mlsanity.com/paper/2402.16142/) (2024)
- [Large Language Models in Wireless Application Design: In-Context Learning-enhanced Automatic Network Intrusion Detection](https://mlsanity.com/paper/2405.11002/) (2024)
- [Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models](https://mlsanity.com/paper/2407.00456/) (2024)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?](https://mlsanity.com/paper/2503.15231/) (2025)
- [Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation](https://mlsanity.com/paper/2409.15895/) (2024)
- [Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones](https://mlsanity.com/paper/2311.15033/) (2023)
- [When Fuzzing Meets LLMs: Challenges and Opportunities](https://mlsanity.com/paper/2404.16297/) (2024)
- [AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instruct](https://mlsanity.com/paper/2405.14906/) (2024)
- [ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design](https://mlsanity.com/paper/2601.21448/) (2026)
- [Combinatorial Optimization for All: Using LLMs to Aid Non-Experts in Improving Optimization Algorithms](https://mlsanity.com/paper/2503.10968/) (2025)
- [From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs](https://mlsanity.com/paper/2601.03808/) (2026)
- [Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective](https://mlsanity.com/paper/2505.14808/) (2025)
- [MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing](https://mlsanity.com/paper/2505.03906/) (2025)
- [Improving Existing Optimization Algorithms with LLMs](https://mlsanity.com/paper/2502.08298/) (2025)
- [Exploring the Challenges and Opportunities of AI-assisted Codebase Generation](https://mlsanity.com/paper/2508.07966/) (2025)
- [ContextModule: Improving Code Completion via Repository-level Contextual Information](https://mlsanity.com/paper/2412.08063/) (2024)
- [Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation](https://mlsanity.com/paper/2506.03535/) (2025)
- [NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification](https://mlsanity.com/paper/2508.02823/) (2025)
- [AI-Driven Self-Evolving Software: A Promising Path Toward Software Automation](https://mlsanity.com/paper/2510.00591/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [Why Human Guidance Matters in Collaborative Vibe Coding](https://mlsanity.com/paper/2602.10473/) (2026)
- [Applying Fine-Tuned LLMs for Reducing Data Needs in Load Profile Analysis](https://mlsanity.com/paper/2406.02479/) (2024)
- [RARe: Retrieval Augmented Retrieval with In-Context Examples](https://mlsanity.com/paper/2410.20088/) (2024)
- [PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework](https://mlsanity.com/paper/2511.03023/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have shown impressive in-context learning (ICL) ability in code generation. LLMs take a prompt consisting of requirement-code examples and a new requirement as input, and output new programs. Existing studies have found that ICL is highly dominated by the examples and thus arises research on example selection. However, existing approaches randomly select examples or only consider the textual similarity of requirements to retrieve, leading to sub-optimal performance. In this paper, we propose a novel learning-based selection approach named LAIL (LLM-Aware In-context Learning) for code generation. Given a candidate example, we exploit LLMs themselves to estimate it by considering the generation probabilities of ground-truth programs given a requirement and the example. We then label candidate examples as positive or negative through the probability feedback. Based on the labeled data, we import a contrastive learning objective to train an effective retriever that acquires the preference of LLMs in code generation. We apply LAIL to three LLMs and evaluate it on three representative datasets (e.g., MBJP, MBPP, and MBCPP). LATA outperforms the state-of-the-art baselines by 11.58%, 6.89%, and 5.07% on CodeGen, and 4.38%, 2.85%, and 2.74% on GPT-3.5 in terms of Pass@1, respectively.