# The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks

arXiv: https://arxiv.org/abs/2402.17679 · HTML: https://mlsanity.com/paper/2402.17679/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2024.

## Built on by (most cited first, 4 of 4)

- [Agentic AI Process Observability: Discovering Behavioral Variability](https://mlsanity.com/paper/2505.20127/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?](https://mlsanity.com/paper/2601.00559/) (2026)
- [LintEval: Benchmarking LLMs on Linter-Guided File Refactoring](https://mlsanity.com/paper/2610.20027/) (2026)

## Abstract (arXiv)

The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.