# Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

arXiv: https://arxiv.org/abs/2310.12357 · HTML: https://mlsanity.com/paper/2310.12357/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2023.

## Built on by (most cited first, 30 of 45)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries](https://mlsanity.com/paper/2405.20653/) (2024)
- [Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning](https://mlsanity.com/paper/2410.04691/) (2024)
- [LAMD: Context-driven Android Malware Detection and Classification with LLMs](https://mlsanity.com/paper/2502.13055/) (2025)
- [On Benchmarking Code LLMs for Android Malware Analysis](https://mlsanity.com/paper/2504.00694/) (2025)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [Rethinking the Evaluation of Secure Code Generation](https://mlsanity.com/paper/2503.15554/) (2025)
- [CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building](https://mlsanity.com/paper/2505.21069/) (2025)
- [Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Framework](https://mlsanity.com/paper/2504.13811/) (2025)
- [The Code Barrier: What LLMs Actually Understand?](https://mlsanity.com/paper/2504.10557/) (2025)
- [Automated Code Review Using Large Language Models at Ericsson: An Experience Report](https://mlsanity.com/paper/2507.19115/) (2025)
- [Revisiting Long-context Modeling from Context Denoising Perspective](https://mlsanity.com/paper/2510.05862/) (2025)
- [The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs](https://mlsanity.com/paper/2504.11711/) (2025)
- [Memory Backdoor Attacks on Neural Networks](https://mlsanity.com/paper/2411.14516/) (2024)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning](https://mlsanity.com/paper/2409.18395/) (2024)
- [Assessing Code Understanding in LLMs](https://mlsanity.com/paper/2504.00065/) (2025)
- [Is Compression Really Linear with Code Intelligence?](https://mlsanity.com/paper/2505.11441/) (2025)
- [Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity](https://mlsanity.com/paper/2512.04355/) (2025)
- [What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Code](https://mlsanity.com/paper/2412.08098/) (2024)
- [Technique Inference Engine: A Recommender Model to Support Cyber Threat Hunting](https://mlsanity.com/paper/2503.04819/) (2025)
- [Uncovering Intention through LLM-Driven Code Snippet Description Generation](https://mlsanity.com/paper/2506.15453/) (2025)
- [FaRAccel: FPGA-Accelerated Defense Architecture for Efficient Bit-Flip Attack Resilience in Transformer Models](https://mlsanity.com/paper/2510.24985/) (2025)
- [Concept Attractors in LLMs and their Applications](https://mlsanity.com/paper/2601.11575/) (2025)
- [Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models](https://mlsanity.com/paper/2604.02490/) (2026)
- [AgentModernize: Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphs](https://mlsanity.com/paper/2605.17535/) (2026)
- [Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection](https://mlsanity.com/paper/2411.09772/) (2024)
- [Regular-pattern-sensitive CRFs for Distant Label Interactions](https://mlsanity.com/paper/2411.12484/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated significant potential in the realm of natural language understanding and programming code processing tasks. Their capacity to comprehend and generate human-like code has spurred research into harnessing LLMs for code analysis purposes. However, the existing body of literature falls short in delivering a systematic evaluation and assessment of LLMs' effectiveness in code analysis, particularly in the context of obfuscated code. This paper seeks to bridge this gap by offering a comprehensive evaluation of LLMs' capabilities in performing code analysis tasks. Additionally, it presents real-world case studies that employ LLMs for code analysis. Our findings indicate that LLMs can indeed serve as valuable tools for automating code analysis, albeit with certain limitations. Through meticulous exploration, this research contributes to a deeper understanding of the potential and constraints associated with utilizing LLMs in code analysis, paving the way for enhanced applications in this critical domain.