mlsanityData to 7 Oct 2026

HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

arXiv 2202.00874 ↗First posted 2 Feb 2022cs.SD cs.AI cs.IRcited by 174

Not on the map. Why?

Abstract (arXiv)

Audio classification is an important task of mapping audio samples into their corresponding labels. Recently, the transformer model with self-attention mechanisms has been adopted in this field. However, existing audio transformers require large GPU memories and long training time, meanwhile relying on pretrained vision models to achieve high performance, which limits the model's scalability in audio tasks. To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time. It is further combined with a token-semantic module to map final outputs into class featuremaps, thus enabling the model for the audio event detection (i.e. localization in time). We evaluate HTS-AT on three datasets of audio classification where it achieves new state-of-the-art (SOTA) results on AudioSet and ESC-50, and equals the SOTA on Speech Command V2. It also achieves better performance in event localization than the previous CNN-based models. Moreover, HTS-AT requires only 35% model parameters and 15% training time of the previous audio transformer. These results demonstrate the high performance and high efficiency of HTS-AT.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

22 of 28 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  8. 2021AST: Audio Spectrogram Transformercited by 634
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2021TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localizationcited by 122
  11. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  12. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
10 more
  1. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
  2. 2021Sound Event Detection: A Tutorialcited by 51
  3. 2018A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labelingcited by 37
  4. 2020MusPy: A Toolkit for Symbolic Music Generationcited by 37
  5. 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29
  6. 2020Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythmcited by 28
  7. 2021Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Datacited by 27
  8. 2021Learning Efficient Representations for Keyword Spotting with Triplet Losscited by 24
  9. 2020Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditionscited by 7
  10. 2020Learning Audio Embeddings with User Listening Data for Content-based Music Recommendationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 43; 2024: 39; 2025: 46; 2026: 33.

174 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  4. 2022Masked Autoencoders that Listencited by 271
  5. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  7. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  10. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  11. 2023Joint Audio and Speech Understandingcited by 99
  12. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
38 more
  1. 2024Long-form music generation with latent diffusioncited by 88
  2. 2023Audio-Visual LLM for Video Understandingcited by 76
  3. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  4. 2024Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversioncited by 58
  5. 2022MAViL: Masked Audio-Video Learnerscited by 56
  6. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  7. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  8. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  9. 2023Separate Anything You Describecited by 43
  10. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  11. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  12. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  13. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  14. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  15. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  16. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  17. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  18. 2024Taming Data and Transformers for Audio Generationcited by 28
  19. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 25
  20. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  21. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  22. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  23. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  24. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  25. 2023Transferable Models for Bioacoustics with Human Language Supervisioncited by 20
  26. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
  27. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  28. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  29. 2023Multiscale Audio Spectrogram Transformer for Efficient Audio Classificationcited by 16
  30. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 15
  31. 2023Learning Tri-modal Embeddings for Zero-Shot Soundscape Mappingcited by 14
  32. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13
  33. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  34. 2024ReCLAP: Improving Zero Shot Audio Classification by Describing Soundscited by 12
  35. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  36. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  37. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  38. 2024MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generationcited by 11

50 most cited of 174 All 174 (JSON)