# DexBERT: Effective, Task-Agnostic and Fine-grained Representation Learning of Android Bytecode

arXiv: https://arxiv.org/abs/2212.05976 · HTML: https://mlsanity.com/paper/2212.05976/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 94 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [graph2vec: Learning Distributed Representations of Graphs](https://mlsanity.com/paper/1707.05005/) (2017)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020)
- [Applying CodeBERT for Automated Program Repair of Java Simple Bugs](https://mlsanity.com/paper/2103.11626/) (2021)
- [Multi-Modal Attention Network Learning for Semantic Source Code Retrieval](https://mlsanity.com/paper/1909.13516/) (2019)
- [DeepTriage: Exploring the Effectiveness of Deep Learning for Bug Triaging](https://mlsanity.com/paper/1801.01275/) (2018)
- [New Alignment Methods for Discriminative Book Summarization](https://mlsanity.com/paper/1305.1319/) (2013)
- [R2-D2: ColoR-inspired Convolutional NeuRal Network (CNN)-based AndroiD Malware Detections](https://mlsanity.com/paper/1705.04448/) (2017)
- [CORE: Automating Review Recommendation for Code Changes](https://mlsanity.com/paper/1912.09652/) (2019)
- [DexRay: A Simple, yet Effective Deep Learning Approach to Android Malware Detection based on Image Representation of Bytecode](https://mlsanity.com/paper/2109.03326/) (2021)
- [A Multi-view Context-aware Approach to Android Malware Detection and Malicious Code Localization](https://mlsanity.com/paper/1704.01759/) (2017)
- [Difuzer: Uncovering Suspicious Hidden Sensitive Operations in Android Apps](https://mlsanity.com/paper/2112.10470/) (2021)
- [The Impact of IR-based Classifier Configuration on the Performance and the Effort of Method-Level Bug Localization](https://mlsanity.com/paper/1806.07727/) (2018)
- [apk2vec: Semi-supervised multi-view representation learning for profiling Android applications](https://mlsanity.com/paper/1809.05693/) (2018)
- [Droidetec: Android Malware Detection and Malicious Code Localization through Deep Learning](https://mlsanity.com/paper/2002.03594/) (2020)
- [Android Malware Detection Using Autoencoder](https://mlsanity.com/paper/1901.07315/) (2019)
- [Using Deep Neural Network for Android Malware Detection](https://mlsanity.com/paper/1904.00736/) (2019)

## Built on by (most cited first, 6 of 6)

- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training](https://mlsanity.com/paper/2502.11191/) (2025)
- [LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learning](https://mlsanity.com/paper/2308.01413/) (2023)
- [DetectBERT: Towards Full App-Level Representation Learning to Detect Android Malware](https://mlsanity.com/paper/2408.16353/) (2024)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysis](https://mlsanity.com/paper/2511.11439/) (2025)

## Abstract (arXiv)

The automation of a large number of software engineering tasks is becoming possible thanks to Machine Learning (ML). Central to applying ML to software artifacts (like source or executable code) is converting them into forms suitable for learning. Traditionally, researchers have relied on manually selected features, based on expert knowledge which is sometimes imprecise and generally incomplete. Representation learning has allowed ML to automatically choose suitable representations and relevant features. Yet, for Android-related tasks, existing models like apk2vec focus on whole-app levels, or target specific tasks like smali2vec, which limits their applicability. Our work is part of a new line of research that investigates effective, task-agnostic, and fine-grained universal representations of bytecode to mitigate both of these two limitations. Such representations aim to capture information relevant to various low-level downstream tasks (e.g., at the class-level). We are inspired by the field of Natural Language Processing, where the problem of universal representation was addressed by building Universal Language Models, such as BERT, whose goal is to capture abstract semantic information about sentences, in a way that is reusable for a variety of tasks. We propose DexBERT, a BERT-like Language Model dedicated to representing chunks of DEX bytecode, the main binary format used in Android applications. We empirically assess whether DexBERT is able to model the DEX language and evaluate the suitability of our model in three distinct class-level software engineering tasks: Malicious Code Localization, Defect Prediction, and Component Type Classification. We also experiment with strategies to deal with the problem of catering to apps having vastly different sizes, and we demonstrate one example of using our technique to investigate what information is relevant to a given task.