# A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

arXiv: https://arxiv.org/abs/2306.07303 · HTML: https://mlsanity.com/paper/2306.07303/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 134 of its 240 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 30 of 134)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)

## Built on by (most cited first, 30 of 71)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Your Transformer is Secretly Linear](https://mlsanity.com/paper/2405.12250/) (2024)
- [Classification, Regression and Segmentation directly from k-Space in Cardiac MRI](https://mlsanity.com/paper/2407.20108/) (2024)
- [Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosis](https://mlsanity.com/paper/2412.02006/) (2024)
- [Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy](https://mlsanity.com/paper/2412.09415/) (2024)
- [STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models](https://mlsanity.com/paper/2502.13119/) (2025)
- [Attention as an RNN](https://mlsanity.com/paper/2405.13956/) (2024)
- [Algebraic Language Models for Inverse Design of Metamaterials via Diffusion Transformers](https://mlsanity.com/paper/2507.15753/) (2025)
- [Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology](https://mlsanity.com/paper/2404.04205/) (2024)
- [T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders](https://mlsanity.com/paper/2506.15881/) (2025)
- [MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection](https://mlsanity.com/paper/2508.10143/) (2025)
- [Differentiable Hierarchical Visual Tokenization](https://mlsanity.com/paper/2511.02652/) (2025)
- [The last Dance : Robust backdoor attack via diffusion models and bayesian approach](https://mlsanity.com/paper/2402.05967/) (2024)
- [FViT: A Focal Vision Transformer with Gabor Filter](https://mlsanity.com/paper/2402.11303/) (2024)
- [Trading Devil: Robust backdoor attack via Stochastic investment models and Bayesian approach](https://mlsanity.com/paper/2406.10719/) (2024)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language](https://mlsanity.com/paper/2410.03718/) (2024)
- [Natural Language Generation in Healthcare: A Review of Methods and Applications](https://mlsanity.com/paper/2505.04073/) (2025)
- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)
- [Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention](https://mlsanity.com/paper/2602.23057/) (2026)
- [PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer](https://mlsanity.com/paper/2404.04886/) (2024)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [Automatic selection of the best neural architecture for time series forecasting](https://mlsanity.com/paper/2501.12215/) (2025)
- [NeuroAMP: A Novel End-to-end General Purpose Deep Neural Amplifier for Personalized Hearing Aids](https://mlsanity.com/paper/2502.10822/) (2025)
- [ChordFormer: A Conformer-Based Architecture for Large-Vocabulary Audio Chord Recognition](https://mlsanity.com/paper/2502.11840/) (2025)
- [Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfaces](https://mlsanity.com/paper/2502.19281/) (2025)
- [Simulation-based inference for Precision Neutrino Physics through Neural Monte Carlo tuning](https://mlsanity.com/paper/2507.23297/) (2025)
- [Hybrid LSTM-Transformer Models for Profiling Highway-Railway Grade Crossings](https://mlsanity.com/paper/2508.00039/) (2025)
- [MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2607.23047/) (2026)

## Abstract (arXiv)

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.