mlsanityData to 14 Oct 2026

How could Neural Networks understand Programs?

arXiv 2105.04297 ↗First posted 10 May 2021cs.PL cs.LG cs.SEcited by 26

Not on the map. Why?

Abstract (arXiv)

Semantic understanding of programs is a fundamental problem for programming language processing (PLP). Recent works that learn representations of code based on pre-training techniques in NLP have pushed the frontiers in this direction. However, the semantics of PL and NL have essential differences. These being ignored, we believe it is difficult to build a model to better understand programs, by either directly applying off-the-shelf NLP pre-training techniques to the source code, or adding features to the model by the heuristic. In fact, the semantics of a program can be rigorously defined by formal semantics in PL theory. For example, the operational semantics, describes the meaning of a valid program as updating the environment (i.e., the memory address-value function) through fundamental operations, such as memory I/O and conditional branching. Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding. To validate our proposal, we present a hierarchical Transformer-based pre-training model called OSCAR to better facilitate the understanding of programs. OSCAR learns from intermediate representation (IR) and an encoded representation derived from static analysis, which are used for representing the fundamental operations and approximating the environment transitions respectively. OSCAR empirically shows the outstanding capability of program semantics understanding on many practical software engineering tasks.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 31 of its 61 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

31 of 61 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  9. 2020On Layer Normalization in the Transformer Architecturecited by 922
  10. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  12. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
19 more
  1. 2020Unsupervised Translation of Programming Languagescited by 263
  2. 2018code2vec: Learning Distributed Representations of Codecited by 250
  3. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  4. 2017Learning to Represent Programs with Graphscited by 220
  5. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  6. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  7. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  8. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  9. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  10. 2020Contrastive Code Representation Learningcited by 76
  11. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  12. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 69
  13. 2018Generative Code Modeling with Graphscited by 60
  14. 2018Neural Code Comprehension: A Learnable Representation of Code Semanticscited by 59
  15. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  16. 2018Aroma: Code Recommendation via Structural Code Searchcited by 29
  17. 2020ProGraML: Graph-based Deep Learning for Program Optimization and Analysiscited by 23
  18. 2020InferCode: Self-Supervised Learning of Code Representations by Predicting Subtreescited by 23
  19. 2020Deep Data Flow Analysiscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 8; 2024: 2; 2025: 4; 2026: 2.

26 papers.

Sort:

By area:

  1. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  2. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  3. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 21
  4. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  5. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  6. 2023Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completioncited by 14
  7. 2022Cross-Language Binary-Source Code Matching with Intermediate Representationscited by 13
  8. 2022A Survey on Pretrained Language Models for Neural Code Intelligencecited by 13
  9. 2023Understanding Programs by Exploiting (Fuzzing) Test Casescited by 13
  10. 2022Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5cited by 10
  11. 2022HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representationscited by 9
  12. 2023Functional Overlap Reranking for Neural Code Generationcited by 9
14 more
  1. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scalecited by 8
  2. 2024CodeArt: Better Code Models by Attention Regularization When Symbols Are Lackingcited by 7
  3. 2022A Neural Network Architecture for Program Understanding Inspired by Human Behaviorscited by 5
  4. 2023Boosting Source Code Learning with Text-Oriented Data Augmentation: An Empirical Studycited by 5
  5. 2025Trace Gadgets: Minimizing Code Context for Machine Learning-Based Vulnerability Predictioncited by 5
  6. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  7. 2024BLens: Contrastive Captioning of Binary Functions using Ensemble Embeddingcited by 3
  8. 2022SAIBench: Benchmarking AI for Sciencecited by 1
  9. 2023VEXIR2Vec: An Architecture-Neutral Embedding Framework for Binary Similaritycited by 1
  10. 2025Can Large Language Models Understand Intermediate Representations in Compilers?cited by 1
  11. 2022Ruby: Unmasking Unsafe Rust in Stripped Binaries via Machine Learningcited by 0
  12. 2025How to Select Pre-Trained Code Models for Reuse? A Learning Perspectivecited by 0
  13. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0
  14. 2026LLM-Based Embeddings for Program Analysis and Optimizationcited by 0