mlsanityData to 14 Oct 2026

ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding

arXiv 2504.00019 ↗First posted 27 Mar 2025cs.CL cs.AI cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) have become a staple of the code-writing toolbox. Their pre-training recipe has, however, remained stagnant over recent years, barring the occasional changes in data sourcing and filtering strategies. In particular, research exploring modifications to Code-LMs' pre-training objectives, geared towards improving data efficiency and better disentangling between syntax and semantics, has been noticeably sparse, especially compared with corresponding efforts in natural language LMs. In this work, we examine grounding on obfuscated code as a means of helping Code-LMs look beyond the surface-form syntax and enhance their pre-training sample efficiency. To this end, we compile ObscuraX, a dataset of approximately 55M source and obfuscated code pairs in seven languages. Subsequently, we pre-train ObscuraCoder models, ranging in size from 255M to 2.8B parameters, on a 272B-token corpus that includes ObscuraX and demonstrate that our obfuscation-based pre-training recipe leads to consistent improvements in Code-LMs' abilities compared to both vanilla autoregressive pre-training as well as existing de-obfuscation (DOBF) objectives. ObscuraCoder demonstrates sizeable gains across multiple tests of syntactic and semantic code understanding, along with improved capabilities in multilingual code completion, multilingual code commit summarization, and multi-purpose library-oriented code generation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 121 of its 164 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

121 of 164 references.

References · 121

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
38 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  6. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  7. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  8. 2023Textbooks Are All You Needcited by 569
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  13. 2023Scaling Data-Constrained Language Modelscited by 403
  14. 2022CodeT: Code Generation with Generated Testscited by 384
  15. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  16. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  17. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  18. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  19. 2021Scaling Laws for Transfercited by 272
  20. 2020Unsupervised Translation of Programming Languagescited by 263
  21. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  22. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  23. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  24. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  25. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  26. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  27. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  28. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  29. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  30. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
  31. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
  32. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  33. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114
  34. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  35. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  36. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  37. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  38. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025textttDroid: A Resource Suite for AI-Generated Code Detectioncited by 9
  2. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  3. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  4. 2026AICD Bench: A Challenging Benchmark for AI-Generated Code Detectioncited by 2