# Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities arXiv: https://arxiv.org/abs/2212.10017 ยท HTML: https://mlsanity.com/paper/2212.10017/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 47 of its 78 references that are in the map. - First posted on arXiv 20 Dec 2022. ## Builds on (most cited first, 30 of 47) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019) - [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020) - [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017) - [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021) - [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022) - [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020) - [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018) - [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019) - [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019) - [CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models](https://mlsanity.com/paper/2206.00052/) (2022) - [What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code](https://mlsanity.com/paper/2202.06840/) (2022) - [A Survey on Deep Learning for Software Engineering](https://mlsanity.com/paper/2011.14597/) (2020) - [What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation](https://mlsanity.com/paper/2007.15356/) (2020) - [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019) - [Straight to the Tree: Constituency Parsing with Neural Syntactic Distance](https://mlsanity.com/paper/1806.04168/) (2018) - [Exploring Software Naturalness through Neural Language Models](https://mlsanity.com/paper/2006.12641/) (2020) ## Built on by (most cited first, 9 of 9) - [Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs](https://mlsanity.com/paper/2305.12138/) (2023) - [AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generation](https://mlsanity.com/paper/2404.16333/) (2024) - [An Empirical Study on Capability of Large Language Models in Understanding Code Semantics](https://mlsanity.com/paper/2407.03611/) (2024) - [Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning](https://mlsanity.com/paper/2404.06201/) (2024) - [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025) - [DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation](https://mlsanity.com/paper/2604.09089/) (2026) - [Can Large Language Models Understand Intermediate Representations in Compilers?](https://mlsanity.com/paper/2502.06854/) (2025) - [Latent Programming Horizons in Coding Agents](https://mlsanity.com/paper/2607.05188/) (2026) - [SwissNYF: Tool Grounded LLM Agents for Black Box Setting](https://mlsanity.com/paper/2402.10051/) (2024) ## Abstract (arXiv) Past research has examined how well these models grasp code syntax, yet their understanding of code semantics still needs to be explored. We extensively analyze seven code models to investigate how code models represent code syntax and semantics. This includes four prominent code pre-trained models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) and three large language models (StarCoder, CodeLlama, and CodeT5+). We have developed four probing tasks to evaluate the models' abilities to learn code syntax and semantics. These tasks focus on reconstructing code syntax and semantic structures-such as AST, CFG, CDG, and DDG - within the models' representation spaces. These structures are fundamental to understanding code. Additionally, we explore the role of syntax tokens in each token representation and the extended dependencies among code tokens. Furthermore, we examine the distribution of attention weights concerning code semantic structures. Through detailed analysis, our results emphasize the strengths and weaknesses of various code models in mastering code syntax and semantics. The findings reveal that these models are proficient in grasping code syntax, effectively capturing the relationships and roles of syntax tokens. However, their ability to encode code semantics shows more variability. This study enriches our understanding of the capabilities of code models in analyzing syntax and semantics. Our findings offer valuable insights for future code model enhancements, helping optimize their application across a range of code-related tasks.