# Transformer Based Machine Fault Detection From Audio Input

arXiv: https://arxiv.org/abs/2604.12733 · HTML: https://mlsanity.com/paper/2604.12733/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection](https://mlsanity.com/paper/1909.09347/) (2019)
- [Statistical Analysis of Nearest Neighbor Methods for Anomaly Detection](https://mlsanity.com/paper/1907.03813/) (2019)
- [Abnormal Respiratory Sound Identification Using Audio-Spectrogram Vision Transformer](https://mlsanity.com/paper/2405.08342/) (2024)

## Abstract (arXiv)

In recent years, Sound AI is being increasingly used to predict machine failures. By attaching a microphone to the machine of interest, one can get real time data on machine behavior from the field. Traditionally, Convolutional Neural Net (CNN) architectures have been used to analyze spectrogram images generated from the sounds captured and predict if the machine is functioning as expected. CNN architectures seem to work well empirically even though they have biases like locality and parameter-sharing which may not be completely relevant for spectrogram analysis. With the successful application of transformer-based models in the field of image processing starting with Vision Transformer (ViT) in 2020, there has been significant interest in leveraging these in the field of Sound AI. Since transformer-based architectures have significantly lower inductive biases, they are expected to perform better than CNNs at spectrogram analysis given enough data. This paper demonstrates the effectiveness of transformer-driven architectures in analyzing Sound data and compares the embeddings they generate with CNNs on the specific task of machine fault detection.