# Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset

arXiv: https://arxiv.org/abs/2207.10664 · HTML: https://mlsanity.com/paper/2207.10664/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 91 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)

## Built on by (most cited first, 12 of 12)

- [Epic-Sounds: A Large-scale Dataset of Actions That Sound](https://mlsanity.com/paper/2302.00646/) (2023)
- [MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understanding](https://mlsanity.com/paper/2306.00576/) (2023)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [Question-Aware Gaussian Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2503.04459/) (2025)
- [AnuraSet: A dataset for benchmarking Neotropical anuran calls identification in passive acoustic monitoring](https://mlsanity.com/paper/2307.06860/) (2023)
- [The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment](https://mlsanity.com/paper/2511.21331/) (2025)
- [Merlin L48 Spectrogram Dataset](https://mlsanity.com/paper/2511.00252/) (2025)
- [Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study](https://mlsanity.com/paper/2605.14031/) (2026)
- [Revisiting Pre-training in Audio-Visual Learning](https://mlsanity.com/paper/2302.03533/) (2023)
- [Not All Birds Look The Same: Identity-Preserving Generation For Birds](https://mlsanity.com/paper/2512.04485/) (2025)
- [Audio-to-Image Bird Species Retrieval without Audio-Image Pairs via Text Distillation](https://mlsanity.com/paper/2602.00681/) (2026)
- [BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment](https://mlsanity.com/paper/2603.23883/) (2026)

## Abstract (arXiv)

We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing research on audiovisual fine-grained categorization. While our community has made great strides in fine-grained visual categorization on images, the counterparts in audio and video fine-grained categorization are relatively unexplored. To encourage advancements in this space, we have carefully constructed the SSW60 dataset to enable researchers to experiment with classifying the same set of categories in three different modalities: images, audio, and video. The dataset covers 60 species of birds and is comprised of images from existing datasets, and brand new, expert-curated audio and video datasets. We thoroughly benchmark audiovisual classification performance and modality fusion experiments through the use of state-of-the-art transformer methods. Our findings show that performance of audiovisual fusion methods is better than using exclusively image or audio based methods for the task of video classification. We also present interesting modality transfer experiments, enabled by the unique construction of SSW60 to encompass three different modalities. We hope the SSW60 dataset and accompanying baselines spur research in this fascinating area.