# Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/1710.10755 · HTML: https://mlsanity.com/paper/1710.10755/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 30 of 34)

- [Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation](https://mlsanity.com/paper/2203.01452/) (2022)
- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation](https://mlsanity.com/paper/2306.03403/) (2023)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Images](https://mlsanity.com/paper/2103.13922/) (2021)
- [Single Frame Semantic Segmentation Using Multi-Modal Spherical Images](https://mlsanity.com/paper/2308.09369/) (2023)
- [Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment](https://mlsanity.com/paper/2305.10983/) (2023)
- [AirSim360: A Panoramic Simulation Platform within Drone View](https://mlsanity.com/paper/2512.02009/) (2025)
- [GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.09115/) (2024)
- [EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning](https://mlsanity.com/paper/2404.10163/) (2024)
- [MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning](https://mlsanity.com/paper/2603.00730/) (2026)
- [Audio-visual Saliency for Omnidirectional Videos](https://mlsanity.com/paper/2311.05190/) (2023)
- [Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model](https://mlsanity.com/paper/2103.15438/) (2021)
- [Panonut360: A Head and Eye Tracking Dataset for Panoramic Video](https://mlsanity.com/paper/2403.17708/) (2024)
- [SHD360: A Benchmark Dataset for Salient Human Detection in 360deg Videos](https://mlsanity.com/paper/2105.11578/) (2021)
- [SalyPath360: Saliency and Scanpath Prediction Framework for Omnidirectional Images](https://mlsanity.com/paper/2201.00096/) (2022)
- [Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization](https://mlsanity.com/paper/2305.02536/) (2023)
- [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024)
- [ASOD60K: An Audio-Induced Salient Object Detection Dataset for Panoramic Videos](https://mlsanity.com/paper/2107.11629/) (2021)
- [MADRL-Based Rate Adaptation for 360deg Video Streaming with Multi-Viewpoint Prediction](https://mlsanity.com/paper/2405.07759/) (2024)
- [Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos](https://mlsanity.com/paper/2508.20221/) (2025)
- [Perceptual Quality Assessment of Virtual Reality Videos in the Wild](https://mlsanity.com/paper/2206.08751/) (2022)
- [Spherical Vision Transformer for 360-degree Video Saliency Prediction](https://mlsanity.com/paper/2308.13004/) (2023)
- [Hierarchical Spherical CNNs with Lifting-based Adaptive Wavelets for Pooling and Unpooling](https://mlsanity.com/paper/2205.15571/) (2022)
- [WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning](https://mlsanity.com/paper/2305.13901/) (2023)
- [Unified Dynamic Scanpath Predictors Outperform Individually Trained Neural Models](https://mlsanity.com/paper/2405.02929/) (2024)
- [A Human-Annotated Video Dataset for Training and Evaluation of 360-Degree Video Summarization Methods](https://mlsanity.com/paper/2406.02991/) (2024)
- [Generating Realistic Synthetic Head Rotation Data for Extended Reality using Deep Learning](https://mlsanity.com/paper/2501.09050/) (2025)

## Abstract (arXiv)

Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.