mlsanityData to 14 Oct 2026

Structure-Aware Fill-in-the-Middle Pretraining for Code

arXiv 2506.00204 ↗First posted 30 May 2025cs.CL cs.AI cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain text and mask random character spans. We propose and evaluate AST-FIM, a pretraining strategy that leverages Abstract Syntax Trees (ASTs) to mask complete syntactic structures at scale, ensuring coherent training examples better aligned with universal code structures and common code editing patterns such as blocks, expressions, or functions. To evaluate real-world fill-in-the-middle (FIM) programming tasks, we introduce Real-FIM-Eval, a benchmark derived from 30,000+ GitHub commits across 12 languages. On infilling tasks, experiments on 1B and 8B parameter models show that AST-FIM is particularly beneficial for real-world code editing as it outperforms standard random-character FIM by up to 5 pts on standard FIM benchmarks. Our code is publicly available at https://github.com/gonglinyuan/ast_fim.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

24 of 27 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
12 more
  1. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  2. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  3. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  4. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  5. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  6. 2023SantaCoder: don't reach for the stars!cited by 170
  7. 2020Enabling Language Models to Fill in the Blankscited by 122
  8. 2024Quantifying Variance in Evaluation Benchmarkscited by 54
  9. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 33
  10. 2025Copilot Arena: A Platform for Code LLM Evaluation in the Wildcited by 26
  11. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
  12. 2024Planning-Aware Code Infilling via Horizon-Length Predictioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  2. 2026Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Modelscited by 2
  3. 2026Hide to Guide: Learning via Semantic Maskingcited by 0
  4. 2026Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigationcited by 0