# Cascade Attention Guided Residue Learning GAN for Cross-Modal Translation

arXiv: https://arxiv.org/abs/1907.01826 · HTML: https://mlsanity.com/paper/1907.01826/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2019.

## Built on by (most cited first, 14 of 14)

- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Cross-View Exocentric to Egocentric Video Synthesis](https://mlsanity.com/paper/2107.03120/) (2021)
- [Continual Attentive Fusion for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2202.00432/) (2022)
- [Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes](https://mlsanity.com/paper/2106.10876/) (2021)
- [Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking](https://mlsanity.com/paper/2112.07423/) (2021)
- [Layout-to-Image Translation with Double Pooling Generative Adversarial Networks](https://mlsanity.com/paper/2108.12900/) (2021)
- [Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses](https://mlsanity.com/paper/2107.05468/) (2021)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [Cross-Modal Contrastive Representation Learning for Audio-to-Image Generation](https://mlsanity.com/paper/2207.12121/) (2022)
- [Automated Side Channel Analysis of Media Software with Manifold Learning](https://mlsanity.com/paper/2112.04947/) (2021)
- [Hierarchical Cross-Attention Network for Virtual Try-On](https://mlsanity.com/paper/2411.15542/) (2024)
- [BioX-Bridge: Model Bridging for Unsupervised Cross-Modal Knowledge Transfer across Biosignals](https://mlsanity.com/paper/2510.02276/) (2025)
- [LARNet: Latent Action Representation for Human Action Synthesis](https://mlsanity.com/paper/2110.10899/) (2021)
- [Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator](https://mlsanity.com/paper/2504.18283/) (2025)

## Abstract (arXiv)

Since we were babies, we intuitively develop the ability to correlate the input from different cognitive sensors such as vision, audio, and text. However, in machine learning, this cross-modal learning is a nontrivial task because different modalities have no homogeneous properties. Previous works discover that there should be bridges among different modalities. From neurology and psychology perspective, humans have the capacity to link one modality with another one, e.g., associating a picture of a bird with the only hearing of its singing and vice versa. Is it possible for machine learning algorithms to recover the scene given the audio signal? In this paper, we propose a novel Cascade Attention-Guided Residue GAN (CAR-GAN), aiming at reconstructing the scenes given the corresponding audio signals. Particularly, we present a residue module to mitigate the gap between different modalities progressively. Moreover, a cascade attention guided network with a novel classification loss function is designed to tackle the cross-modal learning task. Our model keeps the consistency in high-level semantic label domain and is able to balance two different modalities. The experimental results demonstrate that our model achieves the state-of-the-art cross-modal audio-visual generation on the challenging Sub-URMP dataset. Code will be available at https://github.com/tuffr5/CAR-GAN.