# BERT4ETH: A Pre-trained Transformer for Ethereum Fraud Detection

arXiv: https://arxiv.org/abs/2303.18138 · HTML: https://mlsanity.com/paper/2303.18138/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Who Are the Phishers? Phishing Scam Detection on Ethereum via Network Embedding](https://mlsanity.com/paper/1911.09259/) (2019)
- [T-EDGE: Temporal WEighted MultiDiGraph Embedding for Ethereum Transaction Network Analysis](https://mlsanity.com/paper/1905.08038/) (2019)
- [TTAGN: Temporal Transaction Aggregation Graph Network for Ethereum Phishing Scams Detection](https://mlsanity.com/paper/2204.13442/) (2022)
- [Behavior-aware Account De-anonymization on Ethereum Interaction Graph](https://mlsanity.com/paper/2203.09360/) (2022)
- [Learning Role-based Graph Embeddings](https://mlsanity.com/paper/1802.02896/) (2018)
- [Fast Sequence-Based Embedding with Diffusion Graphs](https://mlsanity.com/paper/2001.07463/) (2020)
- [Flash Boys 2.0: Frontrunning, Transaction Reordering, and Consensus Instability in Decentralized Exchanges](https://mlsanity.com/paper/1904.05234/) (2019)
- [Blockchain is Watching You: Profiling and Deanonymizing Ethereum Users](https://mlsanity.com/paper/2005.14051/) (2020)
- [Modeling and Understanding Ethereum Transaction Records via a Complex Network Approach](https://mlsanity.com/paper/2012.15462/) (2020)
- [Identity Inference on Blockchain using Graph Neural Network](https://mlsanity.com/paper/2104.06559/) (2021)
- [Detecting and Quantifying Wash Trading on Decentralized Cryptocurrency Exchanges](https://mlsanity.com/paper/2102.07001/) (2021)
- [Sequence-Based Target Coin Prediction for Cryptocurrency Pump-and-Dump](https://mlsanity.com/paper/2204.12929/) (2022)
- [IcoRating: A Deep-Learning System for Scam ICO Identification](https://mlsanity.com/paper/1803.03670/) (2018)

## Built on by (most cited first, 19 of 19)

- [Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2402.01109/) (2024)
- [Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2405.18641/) (2024)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives](https://mlsanity.com/paper/2310.01152/) (2023)
- [MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPs](https://mlsanity.com/paper/2408.09955/) (2024)
- [Ethereum Fraud Detection via Joint Transaction Language Model and Graph Representation Learning](https://mlsanity.com/paper/2409.07494/) (2024)
- [LMAE4Eth: Generalizable and Robust Ethereum Fraud Detection by Exploring Transaction Semantics and Masked Graph Embedding](https://mlsanity.com/paper/2509.03939/) (2025)
- [Effective Illicit Account Detection on Large Cryptocurrency MultiGraphs](https://mlsanity.com/paper/2309.02460/) (2023)
- [BlockScan: Detecting Anomalies in Blockchain Transactions](https://mlsanity.com/paper/2410.04039/) (2024)
- [Leveraging Deep Learning and Online Source Sentiment for Financial Portfolio Management](https://mlsanity.com/paper/2309.16679/) (2023)
- [Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision](https://mlsanity.com/paper/2408.00641/) (2024)
- [MEGA: Message Passing Neural Networks for Multigraphs with EdGe Attributes](https://mlsanity.com/paper/2412.00241/) (2024)
- [FraudTransformer: Time-Aware GPT for Transaction Fraud Detection](https://mlsanity.com/paper/2509.23712/) (2025)
- [Large Language Model based Smart Contract Auditing with LLMBugScanner](https://mlsanity.com/paper/2512.02069/) (2025)
- [Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects](https://mlsanity.com/paper/2502.04899/) (2025)
- [Linking Cryptoasset Attribution Tags to Knowledge Graph Entities: An LLM-based Approach](https://mlsanity.com/paper/2502.10453/) (2025)
- [NEST: Nested Event Stream Transformer for Sequences of Multisets](https://mlsanity.com/paper/2602.00520/) (2026)
- [MELT: A Behavioral Trace Dataset for High-Risk Memecoin Launch Detection](https://mlsanity.com/paper/2602.13480/) (2026)
- [ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift](https://mlsanity.com/paper/2609.23659/) (2026)

## Abstract (arXiv)

As various forms of fraud proliferate on Ethereum, it is imperative to safeguard against these malicious activities to protect susceptible users from being victimized. While current studies solely rely on graph-based fraud detection approaches, it is argued that they may not be well-suited for dealing with highly repetitive, skew-distributed and heterogeneous Ethereum transactions. To address these challenges, we propose BERT4ETH, a universal pre-trained Transformer encoder that serves as an account representation extractor for detecting various fraud behaviors on Ethereum. BERT4ETH features the superior modeling capability of Transformer to capture the dynamic sequential patterns inherent in Ethereum transactions, and addresses the challenges of pre-training a BERT model for Ethereum with three practical and effective strategies, namely repetitiveness reduction, skew alleviation and heterogeneity modeling. Our empirical evaluation demonstrates that BERT4ETH outperforms state-of-the-art methods with significant enhancements in terms of the phishing account detection and de-anonymization tasks. The code for BERT4ETH is available at: https://github.com/git-disl/BERT4ETH.