# The Role of Transformer Models in Advancing Blockchain Technology: A Systematic Survey

arXiv: https://arxiv.org/abs/2409.02139 · HTML: https://mlsanity.com/paper/2409.02139/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 201 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [Securify: Practical Security Analysis of Smart Contracts](https://mlsanity.com/paper/1806.01143/) (2018)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [Combining Graph Neural Networks with Expert Knowledge for Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2107.11598/) (2021)
- [Security and Privacy on Blockchain](https://mlsanity.com/paper/1903.07602/) (2019)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [Tree-to-Sequence Attentional Neural Machine Translation](https://mlsanity.com/paper/1603.06075/) (2016)
- [Finding The Greedy, Prodigal, and Suicidal Contracts at Scale](https://mlsanity.com/paper/1802.06038/) (2018)
- [Neural Network Inference on Mobile SoCs](https://mlsanity.com/paper/1908.11450/) (2019)
- [A Multi-Modal Transformer-based Code Summarization Approach for Smart Contracts](https://mlsanity.com/paper/2103.07164/) (2021)
- [Source Code Summarization with Structural Relative Position Guided Transformer](https://mlsanity.com/paper/2202.06521/) (2022)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [A Survey on the Security of Blockchain Systems](https://mlsanity.com/paper/1802.06993/) (2018)
- [Mining Fix Patterns for FindBugs Violations](https://mlsanity.com/paper/1712.03201/) (2017)

## Built on by (most cited first, 3 of 3)

- [Sentra-Guard: A Real-Time Multilingual Defense Against Adversarial LLM Prompts](https://mlsanity.com/paper/2510.22628/) (2025)
- [DecoKAN: Interpretable Decomposition for Forecasting Cryptocurrency Market Dynamics](https://mlsanity.com/paper/2512.20028/) (2025)
- [Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models](https://mlsanity.com/paper/2607.27350/) (2026)

## Abstract (arXiv)

As blockchain technology rapidly evolves, the demand for enhanced efficiency, security, and scalability grows.Transformer models, as powerful deep learning architectures,have shown unprecedented potential in addressing various blockchain challenges. However, a systematic review of Transformer applications in blockchain is lacking. This paper aims to fill this research gap by surveying over 200 relevant papers, comprehensively reviewing practical cases and research progress of Transformers in blockchain applications. Our survey covers key areas including anomaly detection, smart contract security analysis, cryptocurrency prediction and trend analysis, and code summary generation. To clearly articulate the advancements of Transformers across various blockchain domains, we adopt a domain-oriented classification system, organizing and introducing representative methods based on major challenges in current blockchain research. For each research domain,we first introduce its background and objectives, then review previous representative methods and analyze their limitations,and finally introduce the advancements brought by Transformer models. Furthermore, we explore the challenges of utilizing Transformer, such as data privacy, model complexity, and real-time processing requirements. Finally, this article proposes future research directions, emphasizing the importance of exploring the Transformer architecture in depth to adapt it to specific blockchain applications, and discusses its potential role in promoting the development of blockchain technology. This review aims to provide new perspectives and a research foundation for the integrated development of blockchain technology and machine learning, supporting further innovation and application expansion of blockchain technology.