# Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?

arXiv: https://arxiv.org/abs/2401.01512 · HTML: https://mlsanity.com/paper/2401.01512/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2024.

## Built on by (most cited first, 6 of 6)

- [How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study](https://mlsanity.com/paper/2412.18989/) (2024)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [SnipGen: A Mining Repository Framework for Evaluating LLMs for Code](https://mlsanity.com/paper/2502.07046/) (2025)
- [Bringing Structure to Naturalness: On the Naturalness of ASTs](https://mlsanity.com/paper/2504.08234/) (2025)
- [Toward Neurosymbolic Program Comprehension](https://mlsanity.com/paper/2502.01806/) (2025)
- [How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?](https://mlsanity.com/paper/2512.15468/) (2025)

## Abstract (arXiv)

This paper discusses the limitations of evaluating Masked Language Models (MLMs) in code completion tasks. We highlight that relying on accuracy-based measurements may lead to an overestimation of models' capabilities by neglecting the syntax rules of programming languages. To address these issues, we introduce a technique called SyntaxEval in which Syntactic Capabilities are used to enhance the evaluation of MLMs. SyntaxEval automates the process of masking elements in the model input based on their Abstract Syntax Trees (ASTs). We conducted a case study on two popular MLMs using data from GitHub repositories. Our results showed negative causal effects between the node types and MLMs' accuracy. We conclude that MLMs under study fail to predict some syntactic capabilities.