mlsanityData to 7 Oct 2026

DOBF: A Deobfuscation Pre-Training Objective for Programming Languages

arXiv 2102.07492 ↗First posted 15 Feb 2021cs.CLcited by 68

Not on the map. Why?

Abstract (arXiv)

Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

40 of 62 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
28 more
  1. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  2. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  3. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  4. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  5. 2020Unsupervised Translation of Programming Languagescited by 262
  6. 2018code2vec: Learning Distributed Representations of Codecited by 249
  7. 2017Learning to Represent Programs with Graphscited by 219
  8. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  9. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  10. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  11. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  12. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  13. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  14. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  15. 2020Contrastive Code Representation Learningcited by 76
  16. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  17. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  18. 2017Code Completion with Neural Attention and Pointer Networkscited by 49
  19. 2019When Deep Learning Met Code Searchcited by 45
  20. 2020Fast and Memory-Efficient Neural Code Completioncited by 38
  21. 2019DIRE: A Neural Approach to Decompiled Identifier Namingcited by 36
  22. 2014Learning Natural Coding Conventionscited by 28
  23. 2018A General Path-Based Representation for Predicting Program Propertiescited by 28
  24. 2019Learning to Fix Build Errors with Graph2Diff Neural Networkscited by 26
  25. 2017Dynamic Neural Program Embedding for Program Repaircited by 23
  26. 2019A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learningcited by 18
  27. 2018Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contextscited by 11
  28. 2020Industry-scale IR-based Bug Localization: A Perspective from Facebookcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 16; 2023: 23; 2024: 12; 2025: 7; 2026: 3.

68 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  4. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  6. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  7. 2023SantaCoder: don't reach for the stars!cited by 169
  8. 2022Natural Language to Code Translation with Executioncited by 122
  9. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  10. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  11. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  12. 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
38 more
  1. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 48
  2. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  3. 2022Code Translation with Compiler Representationscited by 42
  4. 2024LLM4Decompile: Decompiling Binary Code with Large Language Modelscited by 38
  5. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 33
  6. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  7. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 32
  8. 2024Code Representation Learning At Scalecited by 26
  9. 2021Unsupervised Neural Machine Translation with Generative Language Models Onlycited by 24
  10. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  11. 2023CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Executioncited by 21
  12. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 20
  13. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  14. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 15
  15. 2023A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and challengescited by 14
  16. 2022A Survey on Pretrained Language Models for Neural Code Intelligencecited by 13
  17. 2022Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languagescited by 12
  18. 2023Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanationscited by 12
  19. 2023Program Translation via Code Distillationcited by 10
  20. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 9
  21. 2023Exploiting Code Symmetries for Learning Program Semanticscited by 9
  22. 2023Data Augmentation for Code Translation with Comparable Corpora and Multiple Referencescited by 9
  23. 2024Output Format Biases in the Evaluation of Large Language Models for Code Translationcited by 9
  24. 2021SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnosticscited by 8
  25. 2021Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generationcited by 8
  26. 2023Syntax and Domain Aware Model for Unsupervised Program Translationcited by 7
  27. 2024TransAgent: Enhancing LLM-Based Code Translation via Fine-Grained Execution Alignmentcited by 7
  28. 2022On The Cross-Modal Transfer from Natural Language to Code through Adapter Modulescited by 6
  29. 2022Parameter-Efficient Finetuning of Transformers for Source Codecited by 6
  30. 2024Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehensioncited by 6
  31. 2023Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectivescited by 5
  32. 2024Building A Coding Assistant via the Retrieval-Augmented Language Modelcited by 5
  33. 2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
  34. 2024A Joint Learning Model with Variational Interaction for Multilingual Program Translationcited by 4
  35. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  36. 2022AdaptivePaste: Code Adaptation through Learning Semantics-aware Variable Usage Representationscited by 3
  37. 2023Neural Machine Translation for Code Generationcited by 3
  38. 2023Automatically Testing Functional Properties of Code Translation Modelscited by 3

50 most cited of 68 All 68 (JSON)