# Better patching using LLM prompting, via Self-Consistency

arXiv: https://arxiv.org/abs/2306.00108 · HTML: https://mlsanity.com/paper/2306.00108/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 31 May 2023.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation](https://mlsanity.com/paper/1812.08693/) (2018)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Impact of Code Language Models on Automated Program Repair](https://mlsanity.com/paper/2302.05020/) (2023)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [NatGen: Generative pre-training by "Naturalizing" source code](https://mlsanity.com/paper/2206.07585/) (2022)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code](https://mlsanity.com/paper/2206.01335/) (2022)
- [On Multi-Modal Learning of Editing Source Code](https://mlsanity.com/paper/2108.06645/) (2021)

## Built on by (most cited first, 29 of 29)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification](https://mlsanity.com/paper/2502.02747/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [Aligning the Objective of LLM-based Program Repair](https://mlsanity.com/paper/2404.08877/) (2024)
- [Studying LLM Performance on Closed- and Open-source Data](https://mlsanity.com/paper/2402.15100/) (2024)
- [VELO: A Vector Database-Assisted Cloud-Edge Collaborative LLM QoS Optimization Framework](https://mlsanity.com/paper/2406.13399/) (2024)
- [Language Models are Better Bug Detector Through Code-Pair Classification](https://mlsanity.com/paper/2311.07957/) (2023)
- [Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs](https://mlsanity.com/paper/2405.13932/) (2024)
- [COSMosFL: Ensemble of Small Language Models for Fault Localisation](https://mlsanity.com/paper/2502.02908/) (2025)
- [SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioning](https://mlsanity.com/paper/2506.00835/) (2025)
- [AgentSUMO: An Agentic Framework for Interactive Simulation Scenario Generation in SUMO via Large Language Models](https://mlsanity.com/paper/2511.06804/) (2025)
- [Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap](https://mlsanity.com/paper/2604.15075/) (2026)
- [From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap](https://mlsanity.com/paper/2410.20791/) (2024)
- [StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation](https://mlsanity.com/paper/2504.04373/) (2025)
- [VALID-Mol: a Systematic Framework for Validated LLM-Assisted Molecular Design](https://mlsanity.com/paper/2506.23339/) (2025)
- [Towards Adaptive Software Agents for Debugging](https://mlsanity.com/paper/2504.18316/) (2025)
- [What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs](https://mlsanity.com/paper/2509.22796/) (2025)
- [DualResearch: Entropy-Gated Dual-Graph Retrieval for Answer Reconstruction](https://mlsanity.com/paper/2510.08959/) (2025)
- [CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography](https://mlsanity.com/paper/2512.02625/) (2025)
- [T3: Multi-level Tree-based Automatic Program Repair with Large Language Models](https://mlsanity.com/paper/2506.21211/) (2025)
- [DenTab: A Dataset for Table Recognition and Visual QA on Real-World Dental Estimates](https://mlsanity.com/paper/2604.16099/) (2026)
- [Knowledge Graph-Enhanced Zero-Shot Topic Classification: A Multi-Strategy Comparative Study](https://mlsanity.com/paper/2605.30465/) (2026)
- [A Self Consistency Based Reranking for Narrative Question Answering](https://mlsanity.com/paper/2606.15741/) (2026)
- [SFL-MTSC: Leveraging Semantic Frame-Level Multi-Task Self-Consistency for Robust Multi-Intent Spoken Language Understanding](https://mlsanity.com/paper/2606.25552/) (2026)
- [Bulkhead: Automated Semantic Detection and Remediation of Container Escape Vulnerabilities](https://mlsanity.com/paper/2607.12723/) (2026)
- [From Natural Language Policies to Executable Obligations: A Verification Harness for Dependable In-Car LLM Agents](https://mlsanity.com/paper/2608.23282/) (2026)
- [Blockchain-Enabled Artificial Intelligence and AI Agents for Secure Data Sharing and Cybersecurity Applications](https://mlsanity.com/paper/2609.28843/) (2026)

## Abstract (arXiv)

Large Language models (LLMs) can be induced to solve non-trivial problems with "few-shot" prompts including illustrative problem-solution examples. Now if the few-shots also include "chain of thought" (CoT) explanations, which are of the form problem-explanation-solution, LLMs will generate a "explained" solution, and perform even better. Recently an exciting, substantially better technique, self-consistency [1] (S-C) has emerged, based on the intuition that there are many plausible explanations for the right solution; when the LLM is sampled repeatedly to generate a pool of explanation-solution pairs, for a given problem, the most frequently occurring solutions in the pool (ignoring the explanations) tend to be even more likely to be correct! Unfortunately, the use of this highly-performant S-C (or even CoT) approach in software engineering settings is hampered by the lack of explanations; most software datasets lack explanations. In this paper, we describe an application of the S-C approach to program repair, using the commit log on the fix as the explanation, only in the illustrative few-shots. We achieve state-of-the art results, beating previous approaches to prompting-based program repair, on the MODIT dataset; we also find evidence suggesting that the correct commit messages are helping the LLM learn to produce better patches.