# TANet: A new Paradigm for Global Face Super-resolution via Transformer-CNN Aggregation Network

arXiv: https://arxiv.org/abs/2109.08174 · HTML: https://mlsanity.com/paper/2109.08174/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors](https://mlsanity.com/paper/1711.10703/) (2017)
- [Feedback Network for Image Super-Resolution](https://mlsanity.com/paper/1903.09814/) (2019)
- [Learning Spatial Attention for Face Super-Resolution](https://mlsanity.com/paper/2012.01211/) (2020)
- [Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation](https://mlsanity.com/paper/2003.13063/) (2020)
- [Dual-stream Network for Visual Recognition](https://mlsanity.com/paper/2105.14734/) (2021)
- [Deep Learning-based Face Super-Resolution: A Survey](https://mlsanity.com/paper/2101.03749/) (2021)
- [Face Hallucination via Split-Attention in Split-Attention Network](https://mlsanity.com/paper/2010.11575/) (2020)
- [AIM 2020: Scene Relighting and Illumination Estimation Challenge](https://mlsanity.com/paper/2009.12798/) (2020)
- [Multi-scale Self-calibrated Network for Image Light Source Transfer](https://mlsanity.com/paper/2104.08838/) (2021)

## Built on by (most cited first, 2 of 2)

- [Efficient Image Super-Resolution with Feature Interaction Weighted Hybrid Network](https://mlsanity.com/paper/2212.14181/) (2022)
- [W-Net: A Facial Feature-Guided Face Super-Resolution Network](https://mlsanity.com/paper/2406.00676/) (2024)

## Abstract (arXiv)

Recently, face super-resolution (FSR) methods either feed whole face image into convolutional neural networks (CNNs) or utilize extra facial priors (e.g., facial parsing maps, facial landmarks) to focus on facial structure, thereby maintaining the consistency of the facial structure while restoring facial details. However, the limited receptive fields of CNNs and inaccurate facial priors will reduce the naturalness and fidelity of the reconstructed face. In this paper, we propose a novel paradigm based on the self-attention mechanism (i.e., the core of Transformer) to fully explore the representation capacity of the facial structure feature. Specifically, we design a Transformer-CNN aggregation network (TANet) consisting of two paths, in which one path uses CNNs responsible for restoring fine-grained facial details while the other utilizes a resource-friendly Transformer to capture global information by exploiting the long-distance visual relation modeling. By aggregating the features from the above two paths, the consistency of global facial structure and fidelity of local facial detail restoration are strengthened simultaneously. Experimental results of face reconstruction and recognition verify that the proposed method can significantly outperform the state-of-the-art methods.