mlsanityData to 14 Oct 2026

Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities

arXiv 2212.10017 ↗First posted 20 Dec 2022cs.SE cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Past research has examined how well these models grasp code syntax, yet their understanding of code semantics still needs to be explored. We extensively analyze seven code models to investigate how code models represent code syntax and semantics. This includes four prominent code pre-trained models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) and three large language models (StarCoder, CodeLlama, and CodeT5+). We have developed four probing tasks to evaluate the models' abilities to learn code syntax and semantics. These tasks focus on reconstructing code syntax and semantic structures-such as AST, CFG, CDG, and DDG - within the models' representation spaces. These structures are fundamental to understanding code. Additionally, we explore the role of syntax tokens in each token representation and the extended dependencies among code tokens. Furthermore, we examine the distribution of attention weights concerning code semantic structures. Through detailed analysis, our results emphasize the strengths and weaknesses of various code models in mastering code syntax and semantics. The findings reveal that these models are proficient in grasping code syntax, effectively capturing the relationships and roles of syntax tokens. However, their ability to encode code semantics shows more variability. This study enriches our understanding of the capabilities of code models in analyzing syntax and semantics. Our findings offer valuable insights for future code model enhancements, helping optimize their application across a range of code-related tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

47 of 78 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  9. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  10. 2019Graph Transformer Networkscited by 523
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
35 more
  1. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  2. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  3. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  4. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  5. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  6. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  7. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  8. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  9. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 67
  10. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  11. 2019Adversarial Examples for Models of Codecited by 63
  12. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 53
  13. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  14. 2020A Survey on Deep Learning for Software Engineeringcited by 38
  15. 2020What does BERT know about books, movies and music? Probing BERT for Conversational Recommendationcited by 36
  16. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  17. 2018Straight to the Tree: Constituency Parsing with Neural Syntactic Distancecited by 32
  18. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  19. 2019Test Selection for Deep Learning Systemscited by 27
  20. 2022Probing Pretrained Models of Source Codecited by 26
  21. 2019ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Rankingcited by 25
  22. 2020Quantifying the Contextualization of Word Representations with Semantic Class Probingcited by 18
  23. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18
  24. 2020SCELMo: Source Code Embeddings from Language Modelscited by 16
  25. 2020Fractional norms and quasinorms do not help to overcome the curse of dimensionalitycited by 13
  26. 2021GraphCode2Vec: Generic Code Embedding via Lexical and Program Dependence Analysescited by 11
  27. 2020Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matterscited by 9
  28. 2022Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimationcited by 9
  29. 2021Ranking Models in Unlabeled New Environmentscited by 8
  30. 2022TransRepair: Context-aware Program Repair for Compilation Errorscited by 8
  31. 2022Benchmarking Language Models for Code Syntax Understandingcited by 8
  32. 2018Deep Learning in Software Engineeringcited by 6
  33. 2021Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Modelscited by 6
  34. 2022CommitBART: A Large Pre-trained Model for GitHub Commitscited by 6
  35. 2022LaF: Labeling-Free Model Selection for Automated Deep Neural Network Reusingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 2.

9 papers.

  1. 2023Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMscited by 19
  2. 2024AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generationcited by 10
  3. 2024An Empirical Study on Capability of Large Language Models in Understanding Code Semanticscited by 7
  4. 2024Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learningcited by 4
  5. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  6. 2026DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregationcited by 2
  7. 2025Can Large Language Models Understand Intermediate Representations in Compilers?cited by 1
  8. 2026Latent Programming Horizons in Coding Agentscited by 1
  9. 2024SwissNYF: Tool Grounded LLM Agents for Black Box Settingcited by 0