textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time

While recent audio-visual models have demonstrated impressive performance, their robustness to distributional shifts at test-time remains not fully understood. Existing robustness benchmarks mainly focus on single modalities, making them insufficient for thoroughly assessing the robustness of audio-visual models. Motivated by real-world scenarios where shifts can occur textitsimultaneously in both audio and visual modalities, we introduce textttAVROBUSTBENCH, a comprehensive benchmark designed to evaluate the test-time robustness of audio-visual recognition models. textttAVROBUSTBENCH comprises four audio-visual benchmark datasets, textttAUDIOSET-2C, textttVGGSOUND-2C, textttKINETICS-2C, and textttEPICKITCHENS-2C, each incorporating 75 bimodal audio-visual corruptions that are textitco-occurring and textitcorrelated. Through extensive evaluations, we observe that state-of-the-art supervised and self-supervised audio-visual models exhibit declining robustness as corruption severity increases. Furthermore, online test-time adaptation (TTA) methods, on textttVGGSOUND-2C and textttKINETICS-2C, offer minimal improvements in performance under bimodal corruptions. We further propose textttAV2C, a simple TTA approach enabling on-the-fly cross-modal fusion by penalizing high-entropy samples, which achieves improvements on textttVGGSOUND-2C. We hope that textttAVROBUSTBENCH will steer the development of more effective and robust audio-visual TTA approaches. Our code is available hrefhttps://github.com/sarthaxxxxx/AV-C-Robustness-Benchmarkhere.