# The AI Mechanic: Acoustic Vehicle Characterization Neural Networks

arXiv: https://arxiv.org/abs/2205.09667 · HTML: https://mlsanity.com/paper/2205.09667/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 83 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 23 of 23)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Conditional Computation in Neural Networks for faster models](https://mlsanity.com/paper/1511.06297/) (2015)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification](https://mlsanity.com/paper/1608.04363/) (2016)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data](https://mlsanity.com/paper/2006.05919/) (2020)
- [Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge](https://mlsanity.com/paper/1807.05812/) (2018)
- [Data augmentation approaches for improving animal audio classification](https://mlsanity.com/paper/1912.07756/) (2019)
- [Mixup-Based Acoustic Scene Classification Using Multi-Channel Convolutional Neural Network](https://mlsanity.com/paper/1805.07319/) (2018)
- [Deep Sequential Neural Network](https://mlsanity.com/paper/1410.0510/) (2014)
- [An Ontology-Aware Framework for Audio Event Classification](https://mlsanity.com/paper/2001.10048/) (2020)
- [Surveying Off-Board and Extra-Vehicular Monitoring and Progress Towards Pervasive Diagnostics](https://mlsanity.com/paper/2007.03759/) (2020)
- [DeepDrum: An Adaptive Conditional Neural Network](https://mlsanity.com/paper/1809.06127/) (2018)
- [CascadeML: An Automatic Neural Network Architecture Evolution and Training Algorithm for Multi-label Classification](https://mlsanity.com/paper/1904.10551/) (2019)

## Abstract (arXiv)

In a world increasingly dependent on road-based transportation, it is essential to understand vehicles. We introduce the AI mechanic, an acoustic vehicle characterization deep learning system, as an integrated approach using sound captured from mobile devices to enhance transparency and understanding of vehicles and their condition for non-expert users. We develop and implement novel cascading architectures for vehicle understanding, which we define as sequential, conditional, multi-level networks that process raw audio to extract highly-granular insights. To showcase the viability of cascading architectures, we build a multi-task convolutional neural network that predicts and cascades vehicle attributes to enhance fault detection. We train and test these models on a synthesized dataset reflecting more than 40 hours of augmented audio and achieve >92% validation set accuracy on attributes (fuel type, engine configuration, cylinder count and aspiration type). Our cascading architecture additionally achieved 93.6% validation and 86.8% test set accuracy on misfire fault prediction, demonstrating margins of 16.4% / 7.8% and 4.2% / 1.5% improvement over na"ive and parallel baselines. We explore experimental studies focused on acoustic features, data augmentation, feature fusion, and data reliability. Finally, we conclude with a discussion of broader implications, future directions, and application areas for this work.