# Real-Time Position-Aware View Synthesis from Single-View Input

arXiv: https://arxiv.org/abs/2412.14005 · HTML: https://mlsanity.com/paper/2412.14005/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 73 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Cameras as Rays: Pose Estimation via Ray Diffusion](https://mlsanity.com/paper/2402.14817/) (2024)
- [Generalizable Patch-Based Neural Rendering](https://mlsanity.com/paper/2207.10662/) (2022)
- [Vision Transformer for NeRF-Based View Synthesis from a Single Input Image](https://mlsanity.com/paper/2207.05736/) (2022)
- [Learning to Synthesize a 4D RGBD Light Field from a Single Image](https://mlsanity.com/paper/1708.03292/) (2017)
- [Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images](https://mlsanity.com/paper/2205.11733/) (2022)
- [SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting](https://mlsanity.com/paper/2109.01068/) (2021)
- [Simple and Effective Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2204.02960/) (2022)
- [A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality](https://mlsanity.com/paper/1810.08860/) (2018)

## Abstract (arXiv)

Recent advancements in view synthesis have significantly enhanced immersive experiences across various computer graphics and multimedia applications, including telepresence and entertainment. By enabling the generation of new perspectives from a single input view, view synthesis allows users to better perceive and interact with their environment. However, many state-of-the-art methods, while achieving high visual quality, face limitations in real-time performance, which makes them less suitable for live applications where low latency is critical. In this paper, we present a lightweight, position-aware network designed for real-time view synthesis from a single input image and a target camera pose. The proposed framework consists of a Position Aware Embedding, which efficiently maps positional information from the target pose to generate high dimensional feature maps. These feature maps, along with the input image, are fed into a Rendering Network that merges features from dual encoder branches to resolve both high and low level details, producing a realistic new view of the scene. Experimental results demonstrate that our method achieves superior efficiency and visual quality compared to existing approaches, particularly in handling complex translational movements without explicit geometric operations like warping. This work marks a step toward enabling real-time live and interactive telepresence applications.