mlsanityData to 13 Oct 2026

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale

arXiv 2502.16645 ↗First posted 23 Feb 2025cs.CL cs.AI cs.SEcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have exhibited exceptional performance in software engineering yet face challenges in adapting to continually evolving code knowledge, particularly regarding the frequent updates of third-party library APIs. This limitation, stemming from static pre-training datasets, often results in non-executable code or implementations with suboptimal safety and efficiency. To this end, this paper introduces CODESYNC, a data engine for identifying outdated code patterns and collecting real-time code knowledge updates from Python third-party libraries. Building upon CODESYNC, we develop CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Our benchmark offers 3,300 test cases across three evaluation tasks and an update-aware instruction tuning dataset consisting of 2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe that our benchmark can offer a strong foundation for the development of more effective methods for real-time code knowledge updating in the future. The experimental code and dataset are publicly available at: https://github.com/Lucky-voyage/Code-Sync.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 91 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

57 of 91 references.

References · 57

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,450
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,179
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,173
  12. 2022Mass-Editing Memory in a Transformercited by 990
38 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
  2. 2023StarCoder: may the source be with you!cited by 797
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 731
  5. 2023Instruction Tuning with GPT-4cited by 681
  6. 2024A Survey on Large Language Models for Code Generationcited by 656
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 597
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 558
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 497
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 475
  11. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 365
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 359
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 284
  15. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 270
  16. 2018code2vec: Learning Distributed Representations of Codecited by 250
  17. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 243
  18. 2017Learning to Represent Programs with Graphscited by 220
  19. 2021Generating Datasets with Pretrained Language Modelscited by 193
  20. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  21. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  22. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  23. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 100
  24. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 91
  25. 2015Gated Feedback Recurrent Neural Networkscited by 54
  26. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  27. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  28. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 45
  29. 2023Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Studycited by 44
  30. 2016Deep API Learningcited by 41
  31. 2019Multi-Modal Attention Network Learning for Semantic Source Code Retrievalcited by 41
  32. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  33. 2024Task Me Anythingcited by 29
  34. 2021How could Neural Networks understand Programs?cited by 26
  35. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  36. 2024Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessmentcited by 21
  37. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 20
  38. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  2. 2025Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearningcited by 7
  3. 2026To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generationcited by 3
  4. 2026Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agentscited by 2
  5. 2025Understanding Robustness of Model Editing in Code LLMscited by 1
  6. 2026KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generationcited by 1
  7. 2026Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisitioncited by 0
  8. 2026ORCA: Evaluating LLMs on Data Science Code Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution
  2. 2026When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation
  3. 2025RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
  4. 2026Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
  5. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
  6. 2025GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
  7. 2026AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation
  8. 2025ReCode: Updating Code API Knowledge with Reinforcement Learning