# ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio

arXiv: https://arxiv.org/abs/2104.11587 · HTML: https://mlsanity.com/paper/2104.11587/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2021.

## Built on by (most cited first, 24 of 24)

- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025)
- [Sound-Guided Semantic Image Manipulation](https://mlsanity.com/paper/2112.00007/) (2021)
- [AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization](https://mlsanity.com/paper/2210.05060/) (2022)
- [GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation](https://mlsanity.com/paper/2303.10056/) (2023)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models](https://mlsanity.com/paper/2405.02179/) (2024)
- [SemanticAC: Semantics-Assisted Framework for Audio Classification](https://mlsanity.com/paper/2302.05940/) (2023)
- [Underwater-Art: Expanding Information Perspectives With Text Templates For Underwater Acoustic Target Recognition](https://mlsanity.com/paper/2305.19612/) (2023)
- [Adaptive ship-radiated noise recognition with learnable fine-grained wavelet transform](https://mlsanity.com/paper/2306.01002/) (2023)
- [Learning Music Sequence Representation from Text Supervision](https://mlsanity.com/paper/2305.19602/) (2023)
- [Enhanced Multimodal Content Moderation of Children's Videos using Audiovisual Fusion](https://mlsanity.com/paper/2405.06128/) (2024)
- [Effective Audio Classification Network Based on Paired Inverse Pyramid Structure and Dense MLP Block](https://mlsanity.com/paper/2211.02940/) (2022)
- [UFRC: A Unified Framework for Reliable COVID-19 Detection on Crowdsourced Cough Audio](https://mlsanity.com/paper/2204.07763/) (2022)
- [Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning](https://mlsanity.com/paper/2504.21375/) (2025)
- [Leveraging CLIP Encoder for Multimodal Emotion Recognition](https://mlsanity.com/paper/2506.00903/) (2025)
- [SPANER: Shared Prompt Aligner for Multimodal Semantic Representation](https://mlsanity.com/paper/2508.13387/) (2025)
- [End-to-End Audio Strikes Back: Boosting Augmentations Towards An Efficient Audio Classification Network](https://mlsanity.com/paper/2204.11479/) (2022)
- [Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding](https://mlsanity.com/paper/2502.18889/) (2025)
- [Contrastive Learning with Spectrum Information Augmentation in Abnormal Sound Detection](https://mlsanity.com/paper/2509.15570/) (2025)
- [AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning](https://mlsanity.com/paper/2607.15094/) (2026)

## Abstract (arXiv)

Environmental Sound Classification (ESC) is a rapidly evolving field that recently demonstrated the advantages of application of visual domain techniques to the audio-related tasks. Previous studies indicate that the domain-specific modification of cross-domain approaches show a promise in pushing the whole area of ESC forward. In this paper, we present a new time-frequency transformation layer that is based on complex frequency B-spline (fbsp) wavelets. Being used with a high-performance audio classification model, the proposed fbsp-layer provides an accuracy improvement over the previously used Short-Time Fourier Transform (STFT) on standard datasets. We also investigate the influence of different pre-training strategies, including the joint use of two large-scale datasets for weight initialization: ImageNet and AudioSet. Our proposed model out-performs other approaches by achieving accuracies of 95.20 % on the ESC-50 and 89.14 % on the UrbanSound8K datasets. Additionally, we assess the increase of model robustness against additive white Gaussian noise and reduction of an effective sample rate introduced by the proposed layer and demonstrate that the fbsp-layer improves the model's ability to withstand signal perturbations, in comparison to STFT-based training. For the sake of reproducibility, our code is made available.