mlsanityData to 7 Oct 2026

Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization

arXiv 2510.16022 ↗First posted 15 Oct 2025cs.LG cs.SEcited by 1

Not on the map. Why?

Abstract (arXiv)

Adapting pretrained large language models (LLMs) to code domains via supervised fine-tuning (FT) has been commonly used for code generation. However, we identify a previously underappreciated failure mode, the memorization barrier, where strong memorization of downstream code data in the base model could trap optimization and prevent the standard FT from effectively acquiring new, generalizable code knowledge. To overcome this barrier, we propose the information bottleneck (IB)-guided fine-tuning, termed IB-FT, which applies an IB penalty on hidden representations of the code data to compress spurious, memorized features while preserving task-relevant information. Extensive experiments on two code benchmarks (OriGen and Evol-CodeAlpaca-V1) show that IB-FT substantially alleviates the memorization barrier, improves top-1 performance (Pass@1), and yields far more stable gains under the stricter multi-sample metric Pass@k(m) (a problem counts as solved only if at least m of k samples pass unit tests) compared with conventional FT.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

44 of 54 references.

References · 44

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  6. 2016Deep Variational Information Bottleneckcited by 915
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  12. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
32 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  3. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  4. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  5. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  6. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  7. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  8. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  9. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  10. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  11. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  12. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 46
  13. 2024LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluationcited by 46
  14. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  15. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  16. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  17. 2023MFTCoder: Boosting Code LLMs with Multitask Fine-Tuningcited by 34
  18. 2024Test-Driven Development for Code Generationcited by 32
  19. 2023Unveiling Memorization in Code Modelscited by 29
  20. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 28
  21. 2023Traces of Memorisation in Large Language Models for Codecited by 26
  22. 2024Protecting Your LLMs with Information Bottleneckcited by 26
  23. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  24. 2019Efficiently escaping saddle points on manifoldscited by 24
  25. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  26. 2024Learning to Maximize Mutual Information for Chain-of-Thought Distillationcited by 23
  27. 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
  28. 2025Who is using AI to code? Global diffusion and impact of generative AIcited by 17
  29. 2025Calibrating LLMs with Information-Theoretic Evidential Deep Learningcited by 15
  30. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  31. 2025VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validationcited by 11
  32. 2025Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theorycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generationcited by 0