# Spherical Vision Transformer for 360-degree Video Saliency Prediction

arXiv: https://arxiv.org/abs/2308.13004 · HTML: https://mlsanity.com/paper/2308.13004/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 12 of 12)

- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videos](https://mlsanity.com/paper/1806.01320/) (2018)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1710.10755/) (2017)
- [ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos](https://mlsanity.com/paper/2011.10600/) (2020)
- [Assessing Visual Quality of Omnidirectional Videos](https://mlsanity.com/paper/1709.06342/) (2017)
- [SalFBNet: Learning Pseudo-Saliency Distribution via Feedback Convolutional Networks](https://mlsanity.com/paper/2112.03731/) (2021)

## Built on by (most cited first, 2 of 2)

- [Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos](https://mlsanity.com/paper/2508.20221/) (2025)
- [ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefing](https://mlsanity.com/paper/2503.12852/) (2025)

## Abstract (arXiv)

The growing interest in omnidirectional videos (ODVs) that capture the full field-of-view (FOV) has gained 360-degree saliency prediction importance in computer vision. However, predicting where humans look in 360-degree scenes presents unique challenges, including spherical distortion, high resolution, and limited labelled data. We propose a novel vision-transformer-based model for omnidirectional videos named SalViT360 that leverages tangent image representations. We introduce a spherical geometry-aware spatiotemporal self-attention mechanism that is capable of effective omnidirectional video understanding. Furthermore, we present a consistency-based unsupervised regularization term for projection-based 360-degree dense-prediction models to reduce artefacts in the predictions that occur after inverse projection. Our approach is the first to employ tangent images for omnidirectional saliency prediction, and our experimental results on three ODV saliency datasets demonstrate its effectiveness compared to the state-of-the-art.