# Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices

arXiv: https://arxiv.org/abs/2206.00291 · HTML: https://mlsanity.com/paper/2206.00291/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [Attention-guided Network for Ghost-free High Dynamic Range Imaging](https://mlsanity.com/paper/1904.10293/) (2019)
- [Video Super-resolution with Temporal Group Attention](https://mlsanity.com/paper/2007.10595/) (2020)
- [Temporally Distributed Networks for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2004.01800/) (2020)
- [Fast Optical Flow using Dense Inverse Search](https://mlsanity.com/paper/1603.03590/) (2016)
- [Learning Parallax Attention for Stereo Image Super-Resolution](https://mlsanity.com/paper/1903.05784/) (2019)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Progressive Sparse Local Attention for Video object detection](https://mlsanity.com/paper/1903.09126/) (2019)
- [Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution for Mobile Devices](https://mlsanity.com/paper/2105.10288/) (2021)
- [Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule](https://mlsanity.com/paper/2003.03977/) (2020)

## Built on by (most cited first, 1 of 1)

- [AIM 2022 Challenge on Super-Resolution of Compressed Image and Video: Dataset, Methods and Results](https://mlsanity.com/paper/2208.11184/) (2022)

## Abstract (arXiv)

Image alignment, also known as image registration, is a critical block used in many computer vision problems. One of the key factors in alignment is efficiency, as inefficient aligners can cause significant overhead to the overall problem. In the literature, there are some blocks that appear to do the alignment operation, although most do not focus on efficiency. Therefore, an image alignment block which can both work in time and/or space and can work on edge devices would be beneficial for almost all networks dealing with multiple images. Given its wide usage and importance, we propose an efficient, cross-attention-based, multi-purpose image alignment block (XABA) suitable to work within edge devices. Using cross-attention, we exploit the relationships between features extracted from images. To make cross-attention feasible for real-time image alignment problems and handle large motions, we provide a pyramidal block based cross-attention scheme. This also captures local relationships besides reducing memory requirements and number of operations. Efficient XABA models achieve real-time requirements of running above 20 FPS performance on NVIDIA Jetson Xavier with 30W power consumption compared to other powerful computers. Used as a sub-block in a larger network, XABA also improves multi-image super-resolution network performance in comparison to other alignment methods.