# AENet: Learning Deep Audio Features for Video Analysis

arXiv: https://arxiv.org/abs/1701.00599 · HTML: https://mlsanity.com/paper/1701.00599/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2017.

## Built on by (most cited first, 13 of 13)

- [Machine Unlearning: Solutions and Challenges](https://mlsanity.com/paper/2308.07061/) (2023)
- [RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation](https://mlsanity.com/paper/2105.04322/) (2021)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement](https://mlsanity.com/paper/2211.10666/) (2022)
- [MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers](https://mlsanity.com/paper/2308.03741/) (2023)
- [Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion](https://mlsanity.com/paper/2204.12756/) (2022)
- [MAiVAR: Multimodal Audio-Image and Video Action Recognizer](https://mlsanity.com/paper/2209.04780/) (2022)
- [Joint Moment Retrieval and Highlight Detection Via Natural Language Queries](https://mlsanity.com/paper/2305.04961/) (2023)
- [Multimodal Classification: Current Landscape, Taxonomy and Future Directions](https://mlsanity.com/paper/2109.09020/) (2021)
- [Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection](https://mlsanity.com/paper/2110.06126/) (2021)
- [Optimising MFCC parameters for the automatic detection of respiratory diseases](https://mlsanity.com/paper/2408.07522/) (2024)
- [Preserving Temporal Dynamics in Time Series Generation](https://mlsanity.com/paper/2604.27182/) (2026)

## Abstract (arXiv)

We propose a new deep network for audio event recognition, called AENet. In contrast to speech, sounds coming from audio events may be produced by a wide variety of sources. Furthermore, distinguishing them often requires analyzing an extended time period due to the lack of clear sub-word units that are present in speech. In order to incorporate this long-time frequency structure of audio events, we introduce a convolutional neural network (CNN) operating on a large temporal input. In contrast to previous works this allows us to train an audio event detection system end-to-end. The combination of our network architecture and a novel data augmentation outperforms previous methods for audio event detection by 16%. Furthermore, we perform transfer learning and show that our model learnt generic audio features, similar to the way CNNs learn generic features on vision tasks. In video analysis, combining visual features and traditional audio features such as MFCC typically only leads to marginal improvements. Instead, combining visual features with our AENet features, which can be computed efficiently on a GPU, leads to significant performance improvements on action recognition and video highlight detection. In video highlight detection, our audio features improve the performance by more than 8% over visual features alone.