# Audio-Visual Transformer Based Crowd Counting

arXiv: https://arxiv.org/abs/2109.01926 · HTML: https://mlsanity.com/paper/2109.01926/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds](https://mlsanity.com/paper/1808.01050/) (2018)
- [Context-Aware Crowd Counting](https://mlsanity.com/paper/1811.10452/) (2018)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Switching Convolutional Neural Network for Crowd Counting](https://mlsanity.com/paper/1708.00199/) (2017)
- [Fusion of Detected Objects in Text for Visual Question Answering](https://mlsanity.com/paper/1908.05054/) (2019)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Leveraging Unlabeled Data for Crowd Counting by Learning to Rank](https://mlsanity.com/paper/1803.03095/) (2018)
- [Crowd Counting and Density Estimation by Trellis Encoder-Decoder Network](https://mlsanity.com/paper/1903.00853/) (2019)
- [Deep Multimodal Learning for Audio-Visual Speech Recognition](https://mlsanity.com/paper/1501.05396/) (2015)
- [Audio-visual Recognition of Overlapped speech for the LRS2 dataset](https://mlsanity.com/paper/2001.01656/) (2020)
- [HA-CCN: Hierarchical Attention-based Crowd Counting Network](https://mlsanity.com/paper/1907.10255/) (2019)
- [Deep Feature Augmentation for Occluded Image Classification](https://mlsanity.com/paper/2011.00768/) (2020)
- [Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition](https://mlsanity.com/paper/1809.01728/) (2018)
- [ZoomCount: A Zooming Mechanism for Crowd Counting in Static Images](https://mlsanity.com/paper/2002.12256/) (2020)
- [Audio Set classification with attention model: A probabilistic perspective](https://mlsanity.com/paper/1711.00927/) (2017)
- [Ambient Sound Helps: Audiovisual Crowd Counting in Extreme Conditions](https://mlsanity.com/paper/2005.07097/) (2020)
- [Few-Shot Learning by Integrating Spatial and Frequency Representation](https://mlsanity.com/paper/2105.05348/) (2021)

## Built on by (most cited first, 12 of 12)

- [HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection](https://mlsanity.com/paper/2301.03036/) (2023)
- [Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.12333/) (2022)
- [Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets](https://mlsanity.com/paper/2407.19394/) (2024)
- [Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence](https://mlsanity.com/paper/2112.13310/) (2021)
- [Aggregating Global Features into Local Vision Transformer](https://mlsanity.com/paper/2201.12903/) (2022)
- [RGB-T Multi-Modal Crowd Counting Based on Transformer](https://mlsanity.com/paper/2301.03033/) (2023)
- [DenseTrack: Drone-based Crowd Tracking via Density-aware Motion-appearance Synergy](https://mlsanity.com/paper/2407.17272/) (2024)
- [Towards More Effective PRM-based Crowd Counting via A Multi-resolution Fusion and Attention Network](https://mlsanity.com/paper/2112.09664/) (2021)
- [3M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction](https://mlsanity.com/paper/2310.14859/) (2023)
- [Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.14319/) (2022)
- [Progressive Confident Masking Attention Network for Audio-Visual Segmentation](https://mlsanity.com/paper/2406.02345/) (2024)
- [MAFNet: A Multi-Attention Fusion Network for RGB-T Crowd Counting](https://mlsanity.com/paper/2208.06761/) (2022)

## Abstract (arXiv)

Crowd estimation is a very challenging problem. The most recent study tries to exploit auditory information to aid the visual models, however, the performance is limited due to the lack of an effective approach for feature extraction and integration. The paper proposes a new audiovisual multi-task network to address the critical challenges in crowd counting by effectively utilizing both visual and audio inputs for better modalities association and productive feature extraction. The proposed network introduces the notion of auxiliary and explicit image patch-importance ranking (PIR) and patch-wise crowd estimate (PCE) information to produce a third (run-time) modality. These modalities (audio, visual, run-time) undergo a transformer-inspired cross-modality co-attention mechanism to finally output the crowd estimate. To acquire rich visual features, we propose a multi-branch structure with transformer-style fusion in-between. Extensive experimental evaluations show that the proposed scheme outperforms the state-of-the-art networks under all evaluation settings with up to 33.8% improvement. We also analyze and compare the vision-only variant of our network and empirically demonstrate its superiority over previous approaches.