mlsanityData to 14 Oct 2026

WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning

arXiv 2305.13901 ↗First posted 23 May 2023cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

To date, the widely adopted way to perform fixation collection in panoptic video is based on a head-mounted display (HMD), where users' fixations are collected while wearing an HMD to explore the given panoptic scene freely. However, this widely-used data collection method is insufficient for training deep models to accurately predict which regions in a given panoptic are most important when it contains intermittent salient events. The main reason is that there always exist "blind zooms" when using HMD to collect fixations since the users cannot keep spinning their heads to explore the entire panoptic scene all the time. Consequently, the collected fixations tend to be trapped in some local views, leaving the remaining areas to be the "blind zooms". Therefore, fixation data collected using HMD-based methods that accumulate local views cannot accurately represent the overall global importance - the main purpose of fixations - of complex panoptic scenes. To conquer, this paper introduces the auxiliary window with a dynamic blurring (WinDB) fixation collection approach for panoptic video, which doesn't need HMD and is able to well reflect the regional-wise importance degree. Using our WinDB approach, we have released a new PanopticVideo-300 dataset, containing 300 panoptic clips covering over 225 categories. Specifically, since using WinDB to collect fixations is blind zoom free, there exists frequent and intensive "fixation shifting" - a very special phenomenon that has long been overlooked by the previous research - in our new set. Thus, we present an effective fixation shifting network (FishNet) to conquer it. All these new fixation collection tool, dataset, and network could be very potential to open a new age for fixation-related research and applications in 360o environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 91 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

42 of 91 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  3. 2017Deformable Convolutional Networkscited by 1,488
  4. 2018Spherical CNNscited by 456
  5. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 248
  6. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 194
  7. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  8. 2021Visual Saliency Transformercited by 175
  9. 2020Anomaly Detection in Video via Self-Supervised and Multi-Task Learningcited by 148
  10. 2020HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Featurescited by 105
  11. 2016Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Modelcited by 91
  12. 2019Spherical CNNs on Unstructured Gridscited by 79
30 more
  1. 2019xR-EgoPose: Egocentric 3D Human Pose from an HMD Cameracited by 79
  2. 2019Tangent Images for Mitigating Spherical Distortioncited by 79
  3. 2019Motion Guided Attention for Video Salient Object Detectioncited by 72
  4. 2018Revisiting Video Saliency: A Large-scale Benchmark and a New Modelcited by 67
  5. 2018Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videoscited by 66
  6. 2022OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusioncited by 61
  7. 2018Kernel Transformer Networks for Compact Spherical Convolutioncited by 56
  8. 2019Orientation-aware Semantic Segmentation on Icosahedron Spherescited by 53
  9. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  10. 2020STAViS: Spatio-Temporal AudioVisual Saliency Networkcited by 40
  11. 2022BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimationcited by 35
  12. 2017Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approachcited by 34
  13. 2021SphereSR: 360deg Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representationcited by 33
  14. 2018A Memory Network Approach for Story-based Temporal Summarization of 360deg Videoscited by 29
  15. 2021Effective and Efficient Graph Learning for Multi-view Clusteringcited by 28
  16. 2019Simple vs complex temporal recurrences for video saliency predictioncited by 25
  17. 2016Pano2Vid: Automatic Cinematography for Watching 360circ Videoscited by 22
  18. 2017Deep 360 Pilot: Learning a Deep Agent for Piloting through 360deg Sports Videocited by 22
  19. 2017Learning Spherical Convolution for Fast Features from 360deg Imagerycited by 22
  20. 2018Object Detection in Equirectangular Panoramacited by 20
  21. 2018Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video: A Large-Scale Dataset and a Deep Learning Modelcited by 19
  22. 2020ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videoscited by 19
  23. 2021Deep Saliency Prior for Reducing Visual Distractioncited by 17
  24. 2018A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Videocited by 16
  25. 2021Layout-Guided Novel View Synthesis from a Single Indoor Panoramacited by 16
  26. 2022Panoramic Vision Transformer for Saliency Detection in 360deg Videoscited by 15
  27. 2017Making 360circ Video Watchable in 2D: Learning Videography for Click Free Viewingcited by 13
  28. 2022Multi-Projection Fusion and Refinement Network for Salient Object Detection in 360deg Omnidirectional Imagecited by 13
  29. 2022View-aware Salient Object Detection for 360deg Omnidirectional Imagecited by 6
  30. 2019Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Realitycited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarioscited by 4