# GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time arXiv: https://arxiv.org/abs/2403.10147 · HTML: https://mlsanity.com/paper/2403.10147/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 35 references that are in the map. - First posted on arXiv 15 Mar 2024. ## Builds on (most cited first, 30 of 30) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020) - [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020) - [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023) - [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018) - [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021) - [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023) - [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019) - [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021) - [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020) - [Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model](https://mlsanity.com/paper/2311.06214/) (2023) - [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021) - [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022) - [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021) - [COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2312.07504/) (2023) - [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021) - [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://mlsanity.com/paper/2311.12024/) (2023) - [ARF: Artistic Radiance Fields](https://mlsanity.com/paper/2206.06360/) (2022) - [Is Attention All That NeRF Needs?](https://mlsanity.com/paper/2207.13298/) (2022) - [Light Field Neural Rendering](https://mlsanity.com/paper/2112.09687/) (2021) - [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023) - [MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videos](https://mlsanity.com/paper/2212.13056/) (2022) - [DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields](https://mlsanity.com/paper/2303.14478/) (2023) - [RUST: Latent Neural Scene Representations from Unposed Imagery](https://mlsanity.com/paper/2211.14306/) (2022) - [FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow](https://mlsanity.com/paper/2306.00180/) (2023) - [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021) - [DRO: Deep Recurrent Optimizer for Video to Depth](https://mlsanity.com/paper/2103.13201/) (2021) - [3D Reconstruction with Generalizable Neural Fields using Scene Priors](https://mlsanity.com/paper/2309.15164/) (2023) ## Built on by (most cited first, 20 of 20) - [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024) - [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024) - [InstantSplat: Sparse-view Gaussian Splatting in Seconds](https://mlsanity.com/paper/2403.20309/) (2024) - [IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction](https://mlsanity.com/paper/2510.22706/) (2025) - [CityGS-X: A Scalable Architecture for Efficient and Geometrically Accurate Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2503.23044/) (2025) - [PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2412.12096/) (2024) - [SfM-Free 3D Gaussian Splatting via Hierarchical Training](https://mlsanity.com/paper/2412.01553/) (2024) - [Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2503.14537/) (2025) - [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025) - [STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization](https://mlsanity.com/paper/2511.00033/) (2025) - [SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images](https://mlsanity.com/paper/2510.15072/) (2025) - [Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos](https://mlsanity.com/paper/2505.13440/) (2025) - [Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting](https://mlsanity.com/paper/2509.26455/) (2025) - [Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis](https://mlsanity.com/paper/2503.00371/) (2025) - [DynSUP: Dynamic Gaussian Splatting from An Unposed Image Pair](https://mlsanity.com/paper/2412.00851/) (2024) - [Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platform](https://mlsanity.com/paper/2512.08478/) (2025) - [GSCompleter: A Distillation-Free Plugin for Metric-Aware 3D Gaussian Splatting Completion in Seconds](https://mlsanity.com/paper/2604.20155/) (2026) - [Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2606.10656/) (2026) - [GS-Net: Heterogeneous Vehicle Data Reuse via Generalizable Plug-and-Play 3DGS Module](https://mlsanity.com/paper/2409.11307/) (2024) - [ADGaussian: Generalizable Gaussian Splatting for Autonomous Driving via Multi-modal Joint Learning](https://mlsanity.com/paper/2504.00437/) (2025) ## Abstract (arXiv) This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at ≥ 5 FPS and real-time rendering at ≥ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences.