# FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

arXiv: https://arxiv.org/abs/2312.02380 · HTML: https://mlsanity.com/paper/2312.02380/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [A novel time-frequency Transformer based on self-attention mechanism and its application in fault diagnosis of rolling bearings](https://mlsanity.com/paper/2104.09079/) (2021)
- [FaultNet: A Deep Convolutional Neural Network for bearing fault classification](https://mlsanity.com/paper/2010.02146/) (2020)

## Built on by (most cited first, 3 of 3)

- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [Finding One's Bearings in the Hyperparameter Landscape of a Wide-Kernel Convolutional Fault Detector](https://mlsanity.com/paper/2411.15191/) (2024)
- [Spectral Aliasing Pretext: A novel task for Self-Supervised fault diagnosis in rotating machinery](https://mlsanity.com/paper/2608.05705/) (2026)

## Abstract (arXiv)

The growth of global consumption has motivated important applications of deep learning to smart manufacturing and machine health monitoring. In particular, analyzing vibration data offers great potential to extract meaningful insights into predictive maintenance by the detection of bearing faults. Deep learning can be a powerful method to predict these mechanical failures; however, they lack generalizability to new tasks or datasets and require expensive, labeled mechanical data. We address this by presenting a novel self-supervised pretraining and fine-tuning framework based on transformer models. In particular, we investigate different tokenization and data augmentation strategies to reach state-of-the-art accuracies using transformer models. Furthermore, we demonstrate self-supervised masked pretraining for vibration signals and its application to low-data regimes, task adaptation, and dataset adaptation. Pretraining is able to improve performance on scarce, unseen training samples, as well as when fine-tuning on fault classes outside of the pretraining distribution. Furthermore, pretrained transformers are shown to be able to generalize to a different dataset in a few-shot manner. This introduces a new paradigm where models can be pretrained on unlabeled data from different bearings, faults, and machinery and quickly deployed to new, data-scarce applications to suit specific manufacturing needs.