Learning Spatial Pyramid Attentive Pooling in Image Synthesis and Image-to-Image Translation

Image synthesis and image-to-image translation are two important generative learning tasks. Remarkable progress has been made by learning Generative Adversarial Networks (GANs)~citegoodfellow2014generative and cycle-consistent GANs (CycleGANs)~citezhu2017unpaired respectively. This paper presents a method of learning Spatial Pyramid Attentive Pooling (SPAP) which is a novel architectural unit and can be easily integrated into both generators and discriminators in GANs and CycleGANs. The proposed SPAP integrates Atrous spatial pyramid~citechen2018deeplab, a proposed cascade attention mechanism and residual connections~citehe2016deep. It leverages the advantages of the three components to facilitate effective end-to-end generative learning: (i) the capability of fusing multi-scale information by ASPP; (ii) the capability of capturing relative importance between both spatial locations (especially multi-scale context) or feature channels by attention; (iii) the capability of preserving information and enhancing optimization feasibility by residual connections. Coarse-to-fine and fine-to-coarse SPAP are studied and intriguing attention maps are observed in both tasks. In experiments, the proposed SPAP is tested in GANs on the Celeba-HQ-128 dataset~citekarras2017progressive, and tested in CycleGANs on the Image-to-Image translation datasets including the Cityscape dataset~citecordts2016cityscapes, Facade and Aerial Maps dataset~citezhu2017unpaired, both obtaining better performance.