# Panoramic Vision Transformer for Saliency Detection in 360deg Videos

arXiv: https://arxiv.org/abs/2209.08956 · HTML: https://mlsanity.com/paper/2209.08956/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 69 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving](https://mlsanity.com/paper/1905.01489/) (2019)
- [TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.14862/) (2021)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Spherical CNNs on Unstructured Grids](https://mlsanity.com/paper/1901.02039/) (2019)

## Built on by (most cited first, 15 of 15)

- [TULIP: Transformer for Upsampling of LiDAR Point Clouds](https://mlsanity.com/paper/2312.06733/) (2023)
- [ViSAGe: Video-to-Spatial Audio Generation](https://mlsanity.com/paper/2506.12199/) (2025)
- [DarSwin: Distortion Aware Radial Swin Transformer](https://mlsanity.com/paper/2304.09691/) (2023)
- [NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results](https://mlsanity.com/paper/2604.14816/) (2026)
- [PanoNormal: Monocular Indoor 360deg Surface Normal Estimation](https://mlsanity.com/paper/2405.18745/) (2024)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [360U-Former: HDR Illumination Estimation with Panoramic Adapted Vision Transformers](https://mlsanity.com/paper/2410.13566/) (2024)
- [Temporal Saliency Detection Towards Explainable Transformer-based Timeseries Forecasting](https://mlsanity.com/paper/2212.07771/) (2022)
- [Panoramic Image-to-Image Translation](https://mlsanity.com/paper/2304.04960/) (2023)
- [WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning](https://mlsanity.com/paper/2305.13901/) (2023)
- [DriverGaze360: OmniDirectional Driver Attention with Object-Level Guidance](https://mlsanity.com/paper/2512.14266/) (2025)
- [VENI: Variational Encoder for Natural Illumination](https://mlsanity.com/paper/2601.14079/) (2026)
- [HDR Environment Map Estimation with Latent Diffusion Models](https://mlsanity.com/paper/2507.21261/) (2025)
- [TSalV360: A Method and Dataset for Text-driven Saliency Detection in 360-Degrees Videos](https://mlsanity.com/paper/2509.26208/) (2025)
- [OpenSAL360: Open-Source Crowdsourcing Platform for Omnidirectional Video Saliency Collection](https://mlsanity.com/paper/2609.21480/) (2026)

## Abstract (arXiv)

360^circ video saliency detection is one of the challenging benchmarks for 360^circ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360^circ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature. We present a new framework named Panoramic Vision Transformer (PAVER). We design the encoder using Vision Transformer with deformable convolution, which enables us not only to plug pretrained models from normal videos into our architecture without additional modules or finetuning but also to perform geometric approximation only once, unlike previous deep CNN-based approaches. Thanks to its powerful encoder, PAVER can learn the saliency from three simple relative relations among local patch features, outperforming state-of-the-art models for the Wild360 benchmark by large margins without supervision or auxiliary information like class activation. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision, including head movement.