# SplatFormer: Point Transformer for Robust 3D Gaussian Splatting

arXiv: https://arxiv.org/abs/2411.06390 · HTML: https://mlsanity.com/paper/2411.06390/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 67 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 30 of 60)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)

## Built on by (most cited first, 16 of 16)

- [SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting](https://mlsanity.com/paper/2506.08710/) (2025)
- [ReSplat: Learning Recurrent Gaussian Splatting](https://mlsanity.com/paper/2510.08575/) (2025)
- [BevSplat: Resolving Height Ambiguity via Feature-Based Gaussian Primitives for Weakly-Supervised Cross-View Localization](https://mlsanity.com/paper/2502.09080/) (2025)
- [Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2503.14698/) (2025)
- [SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images](https://mlsanity.com/paper/2510.15072/) (2025)
- [EGGS: Exchangeable 2D/3D Gaussian Splatting for Geometry-Appearance Balanced Novel View Synthesis](https://mlsanity.com/paper/2512.02932/) (2025)
- [SurfSplat: Conquering Feedforward 2D Gaussian Splatting with Surface Continuity Priors](https://mlsanity.com/paper/2602.02000/) (2026)
- [CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference](https://mlsanity.com/paper/2508.20375/) (2025)
- [SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion](https://mlsanity.com/paper/2605.01466/) (2026)
- [Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction](https://mlsanity.com/paper/2605.22420/) (2026)
- [GaussTrace: Provenance Analysis of 3D Gaussian Splatting Models with Evidence-based LLM Reasoning](https://mlsanity.com/paper/2606.10612/) (2026)
- [Quantile Rendering: Efficiently Embedding High-dimensional Feature on 3D Gaussian Splatting](https://mlsanity.com/paper/2512.20927/) (2025)
- [UNICA: A Unified Neural Framework for Controllable 3D Avatars](https://mlsanity.com/paper/2604.02799/) (2026)
- [L2D2-GS: Learning to Densify for Feedforward Dynamic Gaussian Scene Reconstruction](https://mlsanity.com/paper/2606.29374/) (2026)
- [InceptionGS: Generative Bootstrapping for Large-Scale Gaussian Splatting under Unstructured View Sampling](https://mlsanity.com/paper/2609.02747/) (2026)
- [CompactSplat: Spatially Adaptive Gaussian Distribution for Feedforward Scene Reconstruction](https://mlsanity.com/paper/2610.13177/) (2026)

## Abstract (arXiv)

3D Gaussian Splatting (3DGS) has recently transformed photorealistic reconstruction, achieving high visual fidelity and real-time performance. However, rendering quality significantly deteriorates when test views deviate from the camera angles used during training, posing a major challenge for applications in immersive free-viewpoint rendering and navigation. In this work, we conduct a comprehensive evaluation of 3DGS and related novel view synthesis methods under out-of-distribution (OOD) test camera scenarios. By creating diverse test cases with synthetic and real-world datasets, we demonstrate that most existing methods, including those incorporating various regularization techniques and data-driven priors, struggle to generalize effectively to OOD views. To address this limitation, we introduce SplatFormer, the first point transformer model specifically designed to operate on Gaussian splats. SplatFormer takes as input an initial 3DGS set optimized under limited training views and refines it in a single forward pass, effectively removing potential artifacts in OOD test views. To our knowledge, this is the first successful application of point transformers directly on 3DGS sets, surpassing the limitations of previous multi-scene training methods, which could handle only a restricted number of input views during inference. Our model significantly improves rendering quality under extreme novel views, achieving state-of-the-art performance in these challenging scenarios and outperforming various 3DGS regularization techniques, multi-scene models tailored for sparse view synthesis, and diffusion-based frameworks.