# A novel time-frequency Transformer based on self-attention mechanism and its application in fault diagnosis of rolling bearings

arXiv: https://arxiv.org/abs/2104.09079 · HTML: https://mlsanity.com/paper/2104.09079/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Gated Feedback Recurrent Neural Networks](https://mlsanity.com/paper/1502.02367/) (2015)

## Built on by (most cited first, 12 of 12)

- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [A Scoping Review of Machine Learning Applications in Power System Protection and Disturbance Management](https://mlsanity.com/paper/2509.09053/) (2025)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [A Hybrid Transformer-Mamba Network for Single Image Deraining](https://mlsanity.com/paper/2409.00410/) (2024)
- [A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2305.14649/) (2023)
- [MCI-GRU: Stock Prediction Model Based on Multi-Head Cross-Attention and Improved GRU](https://mlsanity.com/paper/2410.20679/) (2024)
- [FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification](https://mlsanity.com/paper/2312.02380/) (2023)
- [Survey on Foundation Models for Prognostics and Health Management in Industrial Cyber-Physical Systems](https://mlsanity.com/paper/2312.06261/) (2023)
- [A Survey: Towards Privacy and Security in Mobile Large Language Models](https://mlsanity.com/paper/2509.02411/) (2025)
- [S2S-FDD: Bridging Industrial Time Series and Natural Language for Explainable Zero-shot Fault Diagnosis](https://mlsanity.com/paper/2603.08048/) (2026)
- [Attention-Enhanced Graph Filtering for False Data Injection Attack Detection and Localization](https://mlsanity.com/paper/2601.18981/) (2026)
- [BearingNAS: Obtaining In-Sensor Intelligent Fault Diagnosis Systems for Bearings Using a Laptop](https://mlsanity.com/paper/2607.18287/) (2026)

## Abstract (arXiv)

The scope of data-driven fault diagnosis models is greatly extended through deep learning (DL). However, the classical convolution and recurrent structure have their defects in computational efficiency and feature representation, while the latest Transformer architecture based on attention mechanism has not yet been applied in this field. To solve these problems, we propose a novel time-frequency Transformer (TFT) model inspired by the massive success of vanilla Transformer in sequence processing. Specially, we design a fresh tokenizer and encoder module to extract effective abstractions from the time-frequency representation (TFR) of vibration signals. On this basis, a new end-to-end fault diagnosis framework based on time-frequency Transformer is presented in this paper. Through the case studies on bearing experimental datasets, we construct the optimal Transformer structure and verify its fault diagnosis performance. The superiority of the proposed method is demonstrated in comparison with the benchmark models and other state-of-the-art methods.