# On Benchmarking Code LLMs for Android Malware Analysis

arXiv: https://arxiv.org/abs/2504.00694 · HTML: https://mlsanity.com/paper/2504.00694/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [Explanation as a Watermark: Towards Harmless and Multi-bit Model Ownership Verification via Watermarking Feature Attribution](https://mlsanity.com/paper/2405.04825/) (2024)
- [CCTEST: Testing and Repairing Code Completion Systems](https://mlsanity.com/paper/2208.08289/) (2022)
- [MaMaDroid: Detecting Android Malware by Building Markov Chains of Behavioral Models](https://mlsanity.com/paper/1612.04433/) (2016)
- [Evaluating Explanation Without Ground Truth in Interpretable Machine Learning](https://mlsanity.com/paper/1907.06831/) (2019)
- [AppPoet: Large Language Model based Android malware detection via multi-view prompt engineering](https://mlsanity.com/paper/2404.18816/) (2024)
- [FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysis](https://mlsanity.com/paper/2308.05362/) (2023)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Automatic Detection of Five API Documentation Smells: Practitioners' Perspectives](https://mlsanity.com/paper/2102.08486/) (2021)
- [Enhancing Android Malware Detection: The Influence of ChatGPT on Decision-centric Task](https://mlsanity.com/paper/2410.04352/) (2024)
- [Towards Accurate Labeling of Android Apps for Reliable Malware Detection](https://mlsanity.com/paper/2007.00464/) (2020)

## Built on by (most cited first, 13 of 13)

- [Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference](https://mlsanity.com/paper/2508.09442/) (2025)
- [Rethinking Data Protection in the (Generative) Artificial Intelligence Era](https://mlsanity.com/paper/2507.03034/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs](https://mlsanity.com/paper/2509.03117/) (2025)
- [Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing](https://mlsanity.com/paper/2509.14335/) (2025)
- [Identifying Adversary Tactics and Techniques in Malware Binaries with an LLM Agent](https://mlsanity.com/paper/2602.06325/) (2026)
- [AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research](https://mlsanity.com/paper/2511.13333/) (2025)
- [Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models](https://mlsanity.com/paper/2604.02490/) (2026)
- [Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysis](https://mlsanity.com/paper/2511.11439/) (2025)
- [CHASE: LLM Agents for Dissecting Malicious PyPI Packages](https://mlsanity.com/paper/2601.06838/) (2026)
- [Moirae: A Multimodal Agent Collaborative Framework for Dynamic Android Malware Detection](https://mlsanity.com/paper/2608.27994/) (2026)
- [SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs](https://mlsanity.com/paper/2609.06149/) (2026)
- [ALIBI: Adversarial Legitimacy Injection in Binary Input against LLM Malware Analyzers](https://mlsanity.com/paper/2609.19722/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated strong capabilities in various code intelligence tasks. However, their effectiveness for Android malware analysis remains underexplored. Decompiled Android malware code presents unique challenges for analysis, due to the malicious logic being buried within a large number of functions and the frequent lack of meaningful function names. This paper presents CAMA, a benchmarking framework designed to systematically evaluate the effectiveness of Code LLMs in Android malware analysis. CAMA specifies structured model outputs to support key malware analysis tasks, including malicious function identification and malware purpose summarization. Built on these, it integrates three domain-specific evaluation metrics (consistency, fidelity, and semantic relevance), enabling rigorous stability and effectiveness assessment and cross-model comparison. We construct a benchmark dataset of 118 Android malware samples from 13 families collected in recent years, encompassing over 7.5 million distinct functions, and use CAMA to evaluate four popular open-source Code LLMs. Our experiments provide insights into how Code LLMs interpret decompiled code and quantify the sensitivity to function renaming, highlighting both their potential and current limitations in malware analysis.