# TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens

arXiv: https://arxiv.org/abs/2604.15239 · HTML: https://mlsanity.com/paper/2604.15239/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 58 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation](https://mlsanity.com/paper/2403.14621/) (2024)
- [DepthSplat: Connecting Gaussian Splatting and Depth](https://mlsanity.com/paper/2410.13862/) (2024)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2402.03307/) (2024)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)

## Built on by (most cited first, 15 of 15)

- [Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation](https://mlsanity.com/paper/2607.14203/) (2026)
- [QuerySplat: Decoupling Geometry and Appearance Representations in 3DGS Prediction](https://mlsanity.com/paper/2608.01186/) (2026)
- [Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation](https://mlsanity.com/paper/2604.18468/) (2026)
- [SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis](https://mlsanity.com/paper/2605.07287/) (2026)
- [ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion](https://mlsanity.com/paper/2607.20417/) (2026)
- [InfiniSplat: Implicit Gaussian Decoding for Large-Baseline Monocular View Synthesis](https://mlsanity.com/paper/2608.02437/) (2026)
- [Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering](https://mlsanity.com/paper/2609.38177/) (2026)
- [InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding](https://mlsanity.com/paper/2608.07144/) (2026)
- [LocusGS: Spatially Grounded Tokens for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2608.12825/) (2026)
- [PhGS: Post-Hoc Pruning and Refinement of Single-View Feed-Forward 3D Gaussian Reconstructions](https://mlsanity.com/paper/2609.20623/) (2026)
- [VoxelTTO: Voxel-Aligned Feed-Forward 3D Gaussian Splatting with Test-Time Optimization](https://mlsanity.com/paper/2609.21498/) (2026)
- [S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens](https://mlsanity.com/paper/2610.08978/) (2026)
- [CompactSplat: Spatially Adaptive Gaussian Distribution for Feedforward Scene Reconstruction](https://mlsanity.com/paper/2610.13177/) (2026)
- [CoTL: Compact Token Learning for 3D Scene Representation](https://mlsanity.com/paper/2610.18446/) (2026)
- [InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views](https://mlsanity.com/paper/2610.20341/) (2026)

## Abstract (arXiv)

In this work, we revisit several key design choices of modern Transformer-based approaches for feed-forward 3D Gaussian Splatting (3DGS) prediction. We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal, and instead propose to directly regress 3D mean coordinates using only a self-supervised rendering loss. This formulation allows us to move from the standard encoder-only design to an encoder-decoder architecture with learnable Gaussian tokens, thereby unbinding the number of predicted primitives from input image resolution and number of views. Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies, while naturally supporting efficient test-time optimization in token space without degrading learned priors. TokenGS achieves state-of-the-art feed-forward reconstruction performance on both static and dynamic scenes, producing more regularized geometry and more balanced 3DGS distribution, while seamlessly recovering emergent scene attributes such as static-dynamic decomposition and scene flow.