mlsanityData to 14 Oct 2026

Does Few-Shot Learning Help LLM Performance in Code Synthesis?

arXiv 2412.02906 ↗First posted 3 Dec 2024cs.SE cs.AI cs.CLcited by 6

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models (LLMs) have made significant strides at code generation through improved model design, training, and chain-of-thought. However, prompt-level optimizations remain an important yet under-explored aspect of LLMs for coding. This work focuses on the few-shot examples present in most code generation prompts, offering a systematic study on whether few-shot examples improve LLM's coding capabilities, which few-shot examples have the largest impact, and how to select impactful examples. Our work offers 2 approaches for selecting few-shot examples, a model-free method, CODEEXEMPLAR-FREE, and a model-based method, CODEEXEMPLAR-BASED. The 2 methods offer a trade-off between improved performance and reliance on training data and interpretability. Both methods significantly improve CodeLlama's coding ability across the popular HumanEval+ coding benchmark. In summary, our work provides valuable insights into how to pick few-shot examples in code generation prompts to improve LLM code generation capabilities.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 43 of its 66 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

43 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Qwen2.5-Coder Technical Report18 Sep
  2. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving18 May
  3. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement22 Feb
  4. 2023Structured Chain-of-Thought Prompting for Code Generation11 May

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
27 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  5. 2024Chronos: Learning the Language of Time Seriescited by 759
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  11. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  12. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  13. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  14. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  15. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  16. 2023Large Language Models as Evolutionary Optimizerscited by 124
  17. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  18. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  19. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  20. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 58
  21. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  22. 2023Can Large Language Models Write Good Property-Based Tests?cited by 31
  23. 2024Customizing Language Model Responses with Contrastive In-Context Learningcited by 19
  24. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  25. 2024RAMBO: Enhancing RAG-based Repository-Level Method Body Completioncited by 10
  26. 2024Learning Program Behavioral Models from Synthesized Input-Output Pairscited by 4
  27. 2020Generating Unit Tests for Documentationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Serverscited by 66
  2. 2024The Impact of Prompt Programming on Function-Level Code Generationcited by 13
  3. 2025Instance-level Randomization: Toward More Stable LLM Evaluationscited by 1
  4. 2026Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generationcited by 1
  5. 2025Rethinking Technology Stack Selection with AI Coding Proficiencycited by 0
  6. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving the Ability of Pre-trained Language Model by Imparting Large Language Model's Experience ↗
  2. 2024Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation
  3. 2024Does Prompt Formatting Have Any Impact on LLM Performance?
  4. 2025Training-free LLM Verification via Recycling Few-shot Examples
  5. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  6. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs
  7. 2026Not the Example, but the Process: How Self-Generated Examples Enhance LLM Reasoning
  8. 2024Mastering the Craft of Data Synthesis for CodeLLMs