# Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs

arXiv: https://arxiv.org/abs/2305.12138 · HTML: https://mlsanity.com/paper/2305.12138/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 16 of 16)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code](https://mlsanity.com/paper/2202.06840/) (2022)
- [SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations](https://mlsanity.com/paper/2201.01549/) (2022)
- [Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection](https://mlsanity.com/paper/2212.08108/) (2022)
- [AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models](https://mlsanity.com/paper/2206.11719/) (2022)
- [GraphCode2Vec: Generic Code Embedding via Lexical and Program Dependence Analyses](https://mlsanity.com/paper/2112.01218/) (2021)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [A Simple, Yet Effective Approach to Finding Biases in Code Generation](https://mlsanity.com/paper/2211.00609/) (2022)
- [Rich Specifications for Ethereum Smart Contract Verification](https://mlsanity.com/paper/2104.10274/) (2021)

## Built on by (most cited first, 19 of 19)

- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?](https://mlsanity.com/paper/2411.01414/) (2024)
- [CoCoNUT: Structural Code Understanding does not fall out of a tree](https://mlsanity.com/paper/2501.16456/) (2025)
- [Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning](https://mlsanity.com/paper/2410.19000/) (2024)
- [Generating Software Architecture Description from Source Code using Reverse Engineering and Large Language Model](https://mlsanity.com/paper/2511.05165/) (2025)
- [Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning](https://mlsanity.com/paper/2404.06201/) (2024)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [Policy-Aware Generative AI for Safe, Auditable Data Access Governance](https://mlsanity.com/paper/2510.23474/) (2025)
- [ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs](https://mlsanity.com/paper/2401.14279/) (2024)
- [MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema](https://mlsanity.com/paper/2510.16357/) (2025)
- [SWE-QA: A Dataset and Benchmark for Complex Code Understanding](https://mlsanity.com/paper/2604.24814/) (2026)
- [Can Large Language Models Understand Intermediate Representations in Compilers?](https://mlsanity.com/paper/2502.06854/) (2025)
- [Investigating Execution-Aware Language Models for Code Optimization](https://mlsanity.com/paper/2503.08228/) (2025)
- [(How) Do Large Language Models Understand High-Level Message Sequence Charts?](https://mlsanity.com/paper/2605.13773/) (2026)
- [Semantic Source Code Segmentation using Small and Large Language Models](https://mlsanity.com/paper/2507.08992/) (2025)
- [On Generalization in Agentic Tool Calling: CoreThink Agentic Reasoner and MAVEN Dataset](https://mlsanity.com/paper/2510.22898/) (2025)
- [Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineering](https://mlsanity.com/paper/2608.29016/) (2026)

## Abstract (arXiv)

Code analysis is fundamental in Software Engineering, supporting debugging, optimization, and security assessment. Human developers approach it through syntax parsing, static semantics inference, and dynamic reasoning. Traditional tools are effective but limited by language specificity and weak cross-language generalization. Large language models (LLMs) are promising for code tasks, yet their capabilities for fundamental code analysis remain underexplored. We structure our study around three aspects aligned with human practices: syntax parsing, static semantics inference, and dynamic reasoning. We evaluate 21 state-of-the-art LLMs across nine tasks in four languages (C, Java, Python, Solidity), including AST generation, CFG construction, data dependency, taint analysis, and flaky test reasoning. We apply a three-layer evaluation protocol (automated metrics, expert adjudication, consistency validation) to 3,124 code samples, achieving high inter-rater reliability (Cohen's kappa = 0.844-0.936) and strong human-machine agreement (Gwet's AC1 = 0.500-0.727, F1 = 0.791-0.882). While the best LLMs excel in syntax parsing (AST 90%+, expression matching 84-100%) and show promise in static analysis, their dynamic reasoning remains limited (<70%) with high data-shift sensitivity (per-project F1 varying 0-1.0). This hierarchy holds across model families and scales, suggesting fundamental rather than transient limitations. These findings show how LLMs complement traditional analyzers: they offer cross-language generalization but non-deterministic outputs needing validation, while traditional tools give deterministic guarantees but need language-specific configuration. We contribute a validated evaluation framework with comparison against traditional analyzers (Tree-sitter, Soot, Joern) and task-specific applicability tiers. Benchmark: https://github.com/mathieu0905/llm_code_analysis.git