mlsanityData to 13 Oct 2026

Code Representation Pre-training with Complements from Program Executions

arXiv 2309.09980 ↗First posted 4 Sep 2023cs.SE cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) for natural language processing have been grafted onto programming language modeling for advancing code intelligence. Although it can be represented in the text format, code is syntactically more rigorous in order to be properly compiled or interpreted to perform a desired set of behaviors given any inputs. In this case, existing works benefit from syntactic representations to learn from code less ambiguously in the forms of abstract syntax tree, control-flow graph, etc. However, programs with the same purpose can be implemented in various ways showing different syntactic representations while the ones with similar implementations can have distinct behaviors. Though trivially demonstrated during executions, such semantics about functionality are challenging to be learned directly from code, especially in an unsupervised manner. Hence, in this paper, we propose FuzzPretrain to explore the dynamic information of programs revealed by their test cases and embed it into the feature representations of code as complements. The test cases are obtained with the assistance of a customized fuzzer and are only required during pre-training. FuzzPretrain yielded more than 6%/9% mAP improvements on code search over its counterparts trained with only source code or AST, respectively. Our extensive experimental results show the benefits of learning discriminative code representations with program executions.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

32 of 47 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,473
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,304
20 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,089
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 905
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
  4. 2019Contrastive Representation Distillationcited by 739
  5. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 651
  6. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  7. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 360
  8. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  9. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  10. 2017Learning to Represent Programs with Graphscited by 220
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  12. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  13. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
  14. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 75
  15. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 34
  16. 2023Code Execution with Pre-trained Language Modelscited by 34
  17. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  18. 2023ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learningcited by 16
  19. 2023Understanding Programs by Exploiting (Fuzzing) Test Casescited by 13
  20. 2022Soft-Labeled Contrastive Pre-training for Function-level Code Representationcited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 3.

8 papers.

  1. 2023The Program Testing Ability of Large Language Models for Codecited by 11
  2. 2024UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testingcited by 5
  3. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarkingcited by 4
  4. 2024FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generationcited by 3
  5. 2025Investigating Execution-Aware Language Models for Code Optimizationcited by 1
  6. 2026Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformaticscited by 1
  7. 2026Content Fuzzing for Escaping Information Cocoons on Digital Social Mediacited by 1
  8. 2026Invariant Pretraining for Robust Code Representationscited by 0