# Deep 360 Pilot: Learning a Deep Agent for Piloting through 360deg Sports Video

arXiv: https://arxiv.org/abs/1705.01759 · HTML: https://mlsanity.com/paper/1705.01759/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2017.

## Built on by (most cited first, 22 of 22)

- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop](https://mlsanity.com/paper/2506.10968/) (2025)
- [Panoramic Vision Transformer for Saliency Detection in 360deg Videos](https://mlsanity.com/paper/2209.08956/) (2022)
- [DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction](https://mlsanity.com/paper/2403.01226/) (2024)
- [Deep 360^circ Optical Flow Estimation Based on Multi-Projection Fusion](https://mlsanity.com/paper/2208.00776/) (2022)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Unbiased IoU for Spherical Image Object Detection](https://mlsanity.com/paper/2108.08029/) (2021)
- [SHD360: A Benchmark Dataset for Salient Human Detection in 360deg Videos](https://mlsanity.com/paper/2105.11578/) (2021)
- [ASOD60K: An Audio-Induced Salient Object Detection Dataset for Panoramic Videos](https://mlsanity.com/paper/2107.11629/) (2021)
- [An Integrated System for Spatio-Temporal Summarization of 360-degrees Videos](https://mlsanity.com/paper/2312.02576/) (2023)
- [WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning](https://mlsanity.com/paper/2305.13901/) (2023)
- [360VFI: A Dataset and Benchmark for Omnidirectional Video Frame Interpolation](https://mlsanity.com/paper/2407.14066/) (2024)
- [Switch-a-View: View Selection Learned from Unlabeled In-the-wild Videos](https://mlsanity.com/paper/2412.18386/) (2024)
- [Saliency-based Multiple Region of Interest Detection from a Single 360deg image](https://mlsanity.com/paper/2209.03656/) (2022)
- [FindView: Precise Target View Localization Task for Look Around Agents](https://mlsanity.com/paper/2303.09054/) (2023)
- [A Human-Annotated Video Dataset for Training and Evaluation of 360-Degree Video Summarization Methods](https://mlsanity.com/paper/2406.02991/) (2024)
- [OmniSense: Towards Edge-Assisted Online Analytics for 360-Degree Videos](https://mlsanity.com/paper/2508.14237/) (2025)
- [TSalV360: A Method and Dataset for Text-driven Saliency Detection in 360-Degrees Videos](https://mlsanity.com/paper/2509.26208/) (2025)

## Abstract (arXiv)

Watching a 360deg sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this "360 piloting" task, we propose "deep 360 pilot" -- a deep learning-based agent for piloting through 360deg sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. We use the policy gradient technique to jointly train our pipeline: by minimizing (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward of focusing on a foreground object. To evaluate our method, we build a new 360-Sports video dataset consisting of five sports domains. We train domain-specific agents and achieve the best performance on viewing angle selection accuracy and transition smoothness compared to [51] and other baselines.