mlsanityData to 14 Oct 2026

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

arXiv 2310.12357 ↗First posted 18 Oct 2023cs.SE cs.CRcited by 45

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated significant potential in the realm of natural language understanding and programming code processing tasks. Their capacity to comprehend and generate human-like code has spurred research into harnessing LLMs for code analysis purposes. However, the existing body of literature falls short in delivering a systematic evaluation and assessment of LLMs' effectiveness in code analysis, particularly in the context of obfuscated code. This paper seeks to bridge this gap by offering a comprehensive evaluation of LLMs' capabilities in performing code analysis tasks. Additionally, it presents real-world case studies that employ LLMs for code analysis. Our findings indicate that LLMs can indeed serve as valuable tools for automating code analysis, albeit with certain limitations. Through meticulous exploration, this research contributes to a deeper understanding of the potential and constraints associated with utilizing LLMs in code analysis, paving the way for enhanced applications in this critical domain.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 23; 2026: 12.

45 papers.

Sort:

By area:

  1. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  2. 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 49
  3. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 28
  4. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  5. 2025LAMD: Context-driven Android Malware Detection and Classification with LLMscited by 18
  6. 2025On Benchmarking Code LLMs for Android Malware Analysiscited by 13
  7. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 10
  8. 2025Rethinking the Evaluation of Secure Code Generationcited by 10
  9. 2025CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Buildingcited by 10
  10. 2025Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Frameworkcited by 6
  11. 2025The Code Barrier: What LLMs Actually Understand?cited by 5
  12. 2025Automated Code Review Using Large Language Models at Ericsson: An Experience Reportcited by 5
33 more
  1. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  2. 2025The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMscited by 4
  3. 2024Memory Backdoor Attacks on Neural Networkscited by 3
  4. 2025Trust Me, I Know This Function: Hijacking LLM Static Analysis using Biascited by 3
  5. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  6. 2024Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuningcited by 2
  7. 2025Assessing Code Understanding in LLMscited by 2
  8. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  9. 2025Counting Without Running: Evaluating LLMs' Reasoning About Code Complexitycited by 2
  10. 2024What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Codecited by 1
  11. 2025Technique Inference Engine: A Recommender Model to Support Cyber Threat Huntingcited by 1
  12. 2025Uncovering Intention through LLM-Driven Code Snippet Description Generationcited by 1
  13. 2025FaRAccel: FPGA-Accelerated Defense Architecture for Efficient Bit-Flip Attack Resilience in Transformer Modelscited by 1
  14. 2025Concept Attractors in LLMs and their Applicationscited by 1
  15. 2026Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Modelscited by 1
  16. 2026AgentModernize: Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphscited by 1
  17. 2024Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detectioncited by 0
  18. 2024Regular-pattern-sensitive CRFs for Distant Label Interactionscited by 0
  19. 2025Security smells in infrastructure as code: a taxonomy update beyond the seven sinscited by 0
  20. 2025Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMscited by 0
  21. 2025Beyond Embeddings: Interpretable Feature Extraction for Binary Code Similaritycited by 0
  22. 2025Leveraging Code Cohesion Analysis to Identify Source Code Supply Chain Attackscited by 0
  23. 2025Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysiscited by 0
  24. 2026CHASE: LLM Agents for Dissecting Malicious PyPI Packagescited by 0
  25. 2026An End-to-End Framework for Functionality-Embedded Provenance Graph Construction and Threat Interpretationcited by 0
  26. 2026LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Softwarecited by 0
  27. 2026Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automationcited by 0
  28. 2026Multi-View Decompilation for LLM-Based Malware Classificationcited by 0
  29. 2026CNnotator: LLM-Guided Memory Safety Annotation Synthesiscited by 0
  30. 2026Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generationcited by 0
  31. 2026SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMscited by 0
  32. 2026On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Qualitycited by 0
  33. 2026Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineeringcited by 0