# WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning

arXiv: https://arxiv.org/abs/2305.13901 · HTML: https://mlsanity.com/paper/2305.13901/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 91 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Learning Steerable Filters for Rotation Equivariant CNNs](https://mlsanity.com/paper/1711.07289/) (2017)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Visual Saliency Transformer](https://mlsanity.com/paper/2104.12099/) (2021)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Spherical CNNs on Unstructured Grids](https://mlsanity.com/paper/1901.02039/) (2019)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [Motion Guided Attention for Video Salient Object Detection](https://mlsanity.com/paper/1909.07061/) (2019)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videos](https://mlsanity.com/paper/1806.01320/) (2018)
- [OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion](https://mlsanity.com/paper/2203.00838/) (2022)
- [Kernel Transformer Networks for Compact Spherical Convolution](https://mlsanity.com/paper/1812.03115/) (2018)
- [Orientation-aware Semantic Segmentation on Icosahedron Spheres](https://mlsanity.com/paper/1907.12849/) (2019)
- [Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation](https://mlsanity.com/paper/2203.14668/) (2022)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimation](https://mlsanity.com/paper/2209.02952/) (2022)
- [Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1710.10755/) (2017)
- [SphereSR: 360deg Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation](https://mlsanity.com/paper/2112.06536/) (2021)
- [A Memory Network Approach for Story-based Temporal Summarization of 360deg Videos](https://mlsanity.com/paper/1805.02838/) (2018)
- [Effective and Efficient Graph Learning for Multi-view Clustering](https://mlsanity.com/paper/2108.06734/) (2021)
- [Simple vs complex temporal recurrences for video saliency prediction](https://mlsanity.com/paper/1907.01869/) (2019)
- [Pano2Vid: Automatic Cinematography for Watching 360circ Videos](https://mlsanity.com/paper/1612.02335/) (2016)
- [Deep 360 Pilot: Learning a Deep Agent for Piloting through 360deg Sports Video](https://mlsanity.com/paper/1705.01759/) (2017)

## Built on by (most cited first, 1 of 1)

- [AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios](https://mlsanity.com/paper/2511.21053/) (2025)

## Abstract (arXiv)

To date, the widely adopted way to perform fixation collection in panoptic video is based on a head-mounted display (HMD), where users' fixations are collected while wearing an HMD to explore the given panoptic scene freely. However, this widely-used data collection method is insufficient for training deep models to accurately predict which regions in a given panoptic are most important when it contains intermittent salient events. The main reason is that there always exist "blind zooms" when using HMD to collect fixations since the users cannot keep spinning their heads to explore the entire panoptic scene all the time. Consequently, the collected fixations tend to be trapped in some local views, leaving the remaining areas to be the "blind zooms". Therefore, fixation data collected using HMD-based methods that accumulate local views cannot accurately represent the overall global importance - the main purpose of fixations - of complex panoptic scenes. To conquer, this paper introduces the auxiliary window with a dynamic blurring (WinDB) fixation collection approach for panoptic video, which doesn't need HMD and is able to well reflect the regional-wise importance degree. Using our WinDB approach, we have released a new PanopticVideo-300 dataset, containing 300 panoptic clips covering over 225 categories. Specifically, since using WinDB to collect fixations is blind zoom free, there exists frequent and intensive "fixation shifting" - a very special phenomenon that has long been overlooked by the previous research - in our new set. Thus, we present an effective fixation shifting network (FishNet) to conquer it. All these new fixation collection tool, dataset, and network could be very potential to open a new age for fixation-related research and applications in 360o environments.