# OmniSyn: Synthesizing 360 Videos with Wide-baseline Panoramas

arXiv: https://arxiv.org/abs/2202.08752 · HTML: https://mlsanity.com/paper/2202.08752/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 68 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 30 of 30)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas](https://mlsanity.com/paper/1807.09620/) (2018)
- [StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction](https://mlsanity.com/paper/1807.08865/) (2018)
- [Layer-structured 3D Scene Inference via View Synthesis](https://mlsanity.com/paper/1807.10264/) (2018)
- [MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images](https://mlsanity.com/paper/2008.06534/) (2020)
- [Spherical View Synthesis for Self-Supervised 360 Depth Estimation](https://mlsanity.com/paper/1909.08112/) (2019)
- [One Shot 3D Photography](https://mlsanity.com/paper/2008.12298/) (2020)
- [Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery](https://mlsanity.com/paper/2103.01623/) (2021)
- [360SD-Net: 360deg Stereo Depth Estimation with Learnable Cost Volume](https://mlsanity.com/paper/1911.04460/) (2019)
- [Sat2Vid: Street-view Panoramic Video Synthesis from a Single Satellite Image](https://mlsanity.com/paper/2012.06628/) (2020)
- [SweepNet: Wide-baseline Omnidirectional Depth Estimation](https://mlsanity.com/paper/1902.10904/) (2019)
- [Peeking Behind Objects: Layered Depth Prediction from a Single Image](https://mlsanity.com/paper/1807.08776/) (2018)
- [Pano2Vid: Automatic Cinematography for Watching 360circ Videos](https://mlsanity.com/paper/1612.02335/) (2016)
- [Deep Multi Depth Panoramas for View Synthesis](https://mlsanity.com/paper/2008.01815/) (2020)
- [Learning Independent Object Motion from Unlabelled Stereoscopic Videos](https://mlsanity.com/paper/1901.01971/) (2019)

## Built on by (most cited first, 5 of 5)

- [PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2412.12096/) (2024)
- [PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas](https://mlsanity.com/paper/2306.01531/) (2023)
- [Tiled Multiplane Images for Practical 3D Photography](https://mlsanity.com/paper/2309.14291/) (2023)
- [LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors](https://mlsanity.com/paper/2412.09597/) (2024)
- [The evolution of volumetric video: A survey of smart transcoding and compression approaches](https://mlsanity.com/paper/2411.02095/) (2024)

## Abstract (arXiv)

Immersive maps such as Google Street View and Bing Streetside provide true-to-life views with a massive collection of panoramas. However, these panoramas are only available at sparse intervals along the path they are taken, resulting in visual discontinuities during navigation. Prior art in view synthesis is usually built upon a set of perspective images, a pair of stereoscopic images, or a monocular image, but barely examines wide-baseline panoramas, which are widely adopted in commercial platforms to optimize bandwidth and storage usage. In this paper, we leverage the unique characteristics of wide-baseline panoramas and present OmniSyn, a novel pipeline for 360deg view synthesis between wide-baseline panoramas. OmniSyn predicts omnidirectional depth maps using a spherical cost volume and a monocular skip connection, renders meshes in 360deg images, and synthesizes intermediate views with a fusion network. We demonstrate the effectiveness of OmniSyn via comprehensive experimental results including comparison with the state-of-the-art methods on CARLA and Matterport datasets, ablation studies, and generalization studies on street views. We envision our work may inspire future research for this unheeded real-world task and eventually produce a smoother experience for navigating immersive maps.