# Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction

arXiv: https://arxiv.org/abs/2208.05220 · HTML: https://mlsanity.com/paper/2208.05220/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 10 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Image to Image Translation for Domain Adaptation](https://mlsanity.com/paper/1712.00479/) (2017)
- [Salient Object Detection via Integrity Learning](https://mlsanity.com/paper/2101.07663/) (2021)
- [UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders](https://mlsanity.com/paper/2004.05763/) (2020)
- [RGB-D Salient Object Detection: A Survey](https://mlsanity.com/paper/2008.00230/) (2020)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Review of Visual Saliency Detection with Comprehensive Information](https://mlsanity.com/paper/1803.03391/) (2018)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate](https://mlsanity.com/paper/2107.13469/) (2021)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [RRNet: Relational Reasoning Network with Parallel Multi-scale Attention for Salient Object Detection in Optical Remote Sensing Images](https://mlsanity.com/paper/2110.14223/) (2021)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [Densely Nested Top-Down Flows for Salient Object Detection](https://mlsanity.com/paper/2102.09133/) (2021)
- [DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction](https://mlsanity.com/paper/1905.10693/) (2019)
- [Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM](https://mlsanity.com/paper/1709.06316/) (2017)
- [Rethink Maximum Mean Discrepancy for Domain Adaptation](https://mlsanity.com/paper/2007.00689/) (2020)

## Abstract (arXiv)

Both visual and auditory information are valuable to determine the salient regions in videos. Deep convolution neural networks (CNN) showcase strong capacity in coping with the audio-visual saliency prediction task. Due to various factors such as shooting scenes and weather, there often exists moderate distribution discrepancy between source training data and target testing data. The domain discrepancy induces to performance degradation on target testing data for CNN models. This paper makes an early attempt to tackle the unsupervised domain adaptation problem for audio-visual saliency prediction. We propose a dual domain-adversarial learning algorithm to mitigate the domain discrepancy between source and target data. First, a specific domain discrimination branch is built up for aligning the auditory feature distributions. Then, those auditory features are fused into the visual features through a cross-modal self-attention module. The other domain discrimination branch is devised to reduce the domain discrepancy of visual features and audio-visual correlations implied by the fused audio-visual features. Experiments on public benchmarks demonstrate that our method can relieve the performance degradation caused by domain discrepancy.