# Activating Self-Attention for Multi-Scene Absolute Pose Regression

arXiv: https://arxiv.org/abs/2411.01443 · HTML: https://mlsanity.com/paper/2411.01443/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Modelling Uncertainty in Deep Learning for Camera Relocalization](https://mlsanity.com/paper/1509.05909/) (2015)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [ViTGAN: Training GANs with Vision Transformers](https://mlsanity.com/paper/2107.04589/) (2021)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)
- [DFNet: Enhance Absolute Pose Regression with Direct Feature Matching](https://mlsanity.com/paper/2204.00559/) (2022)
- [Direct-PoseNet: Absolute Pose Regression with Photometric Consistency](https://mlsanity.com/paper/2104.04073/) (2021)
- [Camera Pose Auto-Encoders for Improving Pose Regression](https://mlsanity.com/paper/2207.05530/) (2022)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)

## Built on by (most cited first, 2 of 2)

- [White Aggregation and Restoration for Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2509.13907/) (2025)
- [Disambiguating 2D-3D Correspondences in Gaussian Splatting-based Feature Fields for Visual Localization](https://mlsanity.com/paper/2605.07351/) (2026)

## Abstract (arXiv)

Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.