mlsanityData to 14 Oct 2026

Alignment with Fill-In-the-Middle for Enhancing Code Generation

arXiv 2508.19532 ↗First posted 27 Aug 2025cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

The code generation capabilities of Large Language Models (LLMs) have advanced applications like tool invocation and problem-solving. However, improving performance in code-related tasks remains challenging due to limited training data that is verifiable with accurate test cases. While Direct Preference Optimization (DPO) has shown promise, existing methods for generating test cases still face limitations. In this paper, we propose a novel approach that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases. Additionally, we introduce the Abstract Syntax Tree (AST) splitting and curriculum training method to enhance the DPO training. Our approach demonstrates significant improvements in code generation tasks, as validated by experiments on benchmark datasets such as HumanEval (+), MBPP (+), APPS, LiveCodeBench, and BigCodeBench. Code and data are available at https://github.com/SenseLLM/StructureCoder.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 63 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

50 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning30 Jun
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  3. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases11 Jun

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
35 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  5. 2021Measuring Coding Challenge Competence With APPScited by 902
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  11. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  12. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  13. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  14. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  15. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  16. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  17. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  18. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  19. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  20. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 77
  21. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  22. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
  23. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  24. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  25. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  26. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  27. 2024CodeDPO: Aligning Code Models with Self Generated and Verified Source Codecited by 34
  28. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 33
  29. 2024EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLMcited by 25
  30. 2024Code-Optimise: Self-Generated Preference Data for Correctness and Efficiencycited by 16
  31. 2024ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generationcited by 14
  32. 2025Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Pointscited by 14
  33. 2023Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrievalcited by 5
  34. 2024Empowering Character-level Text Infilling by Eliminating Sub-Tokenscited by 5
  35. 2023Lexicon-Enhanced Self-Supervised Training for Multilingual Dense Retrievalcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  2. 2026MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversationcited by 1
  3. 2026DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learningcited by 0
  4. 2026Function-Level Execution Feedback for Code Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation
  2. 2025Teaching Your Models to Understand Code via Focal Preference Alignment
  3. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
  4. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  5. 2024Aligning CodeLLMs with Direct Preference Optimization
  6. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
  7. 2025Learning to Align Human Code Preferences
  8. 2024DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs