# GiFT: Gibbs Fine-Tuning for Code Generation

arXiv: https://arxiv.org/abs/2502.11466 · HTML: https://mlsanity.com/paper/2502.11466/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs](https://mlsanity.com/paper/2402.16352/) (2024)
- [SelfCodeAlign: Self-Alignment for Code Generation](https://mlsanity.com/paper/2410.24198/) (2024)
- [WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning](https://mlsanity.com/paper/2312.14187/) (2023)
- [Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search](https://mlsanity.com/paper/2401.04514/) (2024)
- [Rethinking Negative Pairs in Code Search](https://mlsanity.com/paper/2310.08069/) (2023)
- [Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling](https://mlsanity.com/paper/2411.00750/) (2024)
- [B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests](https://mlsanity.com/paper/2409.08692/) (2024)
- [DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs](https://mlsanity.com/paper/2411.13611/) (2024)
- [Exploring Representation-Level Augmentation for Code Search](https://mlsanity.com/paper/2210.12285/) (2022)

## Built on by (most cited first, 3 of 3)

- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions](https://mlsanity.com/paper/2510.10666/) (2025)
- [Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coverage](https://mlsanity.com/paper/2609.31688/) (2026)

## Abstract (arXiv)

Training Large Language Models (LLMs) with synthetic data is a prevalent practice in code generation. A key approach is self-training, where LLMs are iteratively trained on self-generated correct code snippets. In this case, the self-generated codes are drawn from a conditional distribution, conditioned on a specific seed description. However, the seed description is not the only valid representation that aligns with its intended meaning. With all valid descriptions and codes forming a joint space, codes drawn from the conditional distribution would lead to an underrepresentation of the full description-code space. As such, we propose Gibbs Fine-Tuning (GiFT), a novel self-training method inspired by Gibbs sampling. GiFT allows self-generated data to be drawn from the marginal distribution of the joint space, thereby mitigating the biases inherent in conditional sampling. We provide a theoretical analysis demonstrating the potential benefits of fine-tuning LLMs with code derived from the marginal distribution. Furthermore, we propose a perplexity-based code selection method to mitigate the imbalanced long-tail distribution of the self-generated codes. Empirical evaluation of two LLMs across four datasets demonstrates that GiFT achieves superior performance, particularly on more challenging benchmarks. Source code is available at https://github.com/Alex-HaochenLi/GiFT.