# Spatial Visibility and Temporal Dynamics: Revolutionizing Field of View Prediction in Adaptive Point Cloud Video Streaming

arXiv: https://arxiv.org/abs/2409.18236 · HTML: https://mlsanity.com/paper/2409.18236/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 41 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [HAVE-FUN: Human Avatar Reconstruction from Few-Shot Unconstrained Images](https://mlsanity.com/paper/2311.15672/) (2023)
- [Very Long Term Field of View Prediction for 360-degree Video Streaming](https://mlsanity.com/paper/1902.01439/) (2019)
- [Force-Aware Interface via Electromyography for Natural VR/AR Interaction](https://mlsanity.com/paper/2210.01225/) (2022)
- [Cocktail Edge Caching: Ride Dynamic Trends of Content Popularity with Ensemble Learning](https://mlsanity.com/paper/2101.05885/) (2021)
- [FSVVD: A Dataset of Full Scene Volumetric Video](https://mlsanity.com/paper/2303.03599/) (2023)
- [LiveVV: Human-Centered Live Volumetric Video Streaming System](https://mlsanity.com/paper/2310.08205/) (2023)
- [Predictive Edge Caching through Deep Mining of Sequential Patterns in User Content Retrievals](https://mlsanity.com/paper/2210.02657/) (2022)
- [Advancements in Translation Accuracy for Stereo Visual-Inertial Initialization](https://mlsanity.com/paper/2405.15082/) (2024)
- [Progressive Frame Patching for FoV-based Point Cloud Video Streaming](https://mlsanity.com/paper/2303.08336/) (2023)

## Abstract (arXiv)

Field-of-View (FoV) adaptive streaming significantly reduces bandwidth requirement of immersive point cloud video (PCV) by only transmitting visible points in a viewer's FoV. The traditional approaches often focus on trajectory-based 6 degree-of-freedom (6DoF) FoV predictions. The predicted FoV is then used to calculate point visibility. Such approaches do not explicitly consider video content's impact on viewer attention, and the conversion from FoV to point visibility is often error-prone and time-consuming. We reformulate the PCV FoV prediction problem from the cell visibility perspective, allowing for precise decision-making regarding the transmission of 3D data at the cell level based on the predicted visibility distribution. We develop a novel spatial visibility and object-aware graph model that leverages the historical 3D visibility data and incorporates spatial perception, neighboring cell correlation, and occlusion information to predict the cell visibility in the future. Our model significantly improves the long-term cell visibility prediction, reducing the prediction MSE loss by up to 50% compared to the state-of-the-art models while maintaining real-time performance (more than 30fps) for point cloud videos with over 1 million points.