mlsanityData to 14 Oct 2026

Memorization Dynamics of Fill-in-the-Middle Pretraining

arXiv 2605.22981 ↗First posted 21 May 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim memorization remains underexplored. We study the memorization dynamics of FIM in a controlled setting by pretraining matched Llama 3.2 models with FIM and standard left-to-right (LTR) objectives on a FineWeb-Gutenberg corpus containing repeated Gutenberg excerpts. With prefix-based probes, FIM more often recovers short or partially matching spans, while LTR more often assigns high confidence to long exact continuations. We observe that verbatim extraction under FIM-training grows approximately linearly with repetitions over the tested range. Evaluating native FIM-format probes reveals that suffix context is not sufficient: verbatim recall under FIM-training remains strongly anchored in prefix context. Our results also show that evaluating only one span length or probing format can miss important nuances in memorization behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

23 of 33 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
11 more
  1. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  3. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  4. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  5. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  6. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  7. 2024Measuring memorization in language models via probabilistic extractioncited by 41
  8. 2025Extracting memorized pieces of (copyrighted) books from open-weight language modelscited by 30
  9. 2021How BPE Affects Memorization in Transformerscited by 29
  10. 2024The Mosaic Memory of Large Language Modelscited by 10
  11. 2025Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Riskscited by 2

Built on by

Not cited yet.