# Splatter a Video: Video Gaussian Representation for Versatile Processing

arXiv: https://arxiv.org/abs/2406.13870 · HTML: https://mlsanity.com/paper/2406.13870/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 63 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 30 of 48)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes](https://mlsanity.com/paper/2312.14937/) (2023)
- [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [DynIBaR: Neural Dynamic Image-Based Rendering](https://mlsanity.com/paper/2211.11082/) (2022)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle](https://mlsanity.com/paper/2312.03431/) (2023)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)

## Built on by (most cited first, 30 of 31)

- [MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second](https://mlsanity.com/paper/2507.10065/) (2025)
- [EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Mesh](https://mlsanity.com/paper/2506.05554/) (2025)
- [Deformable Radial Kernel Splatting](https://mlsanity.com/paper/2412.11752/) (2024)
- [GaussianVideo: Efficient Video Representation via Hierarchical Gaussian Splatting](https://mlsanity.com/paper/2501.04782/) (2025)
- [VeGaS: Video Gaussian Splatting](https://mlsanity.com/paper/2411.11024/) (2024)
- [Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy](https://mlsanity.com/paper/2506.22432/) (2025)
- [AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2601.00796/) (2026)
- [CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splatting](https://mlsanity.com/paper/2505.22854/) (2025)
- [StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams](https://mlsanity.com/paper/2506.08862/) (2025)
- [C4D: 4D Made from 3D through Dual Correspondences](https://mlsanity.com/paper/2510.14960/) (2025)
- [GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting](https://mlsanity.com/paper/2501.12060/) (2025)
- [Seeing World Dynamics in a Nutshell](https://mlsanity.com/paper/2502.03465/) (2025)
- [Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos](https://mlsanity.com/paper/2509.24209/) (2025)
- [Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models](https://mlsanity.com/paper/2511.00503/) (2025)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Progressively Deformable 2D Gaussian Splatting for Video Representation at Arbitrary Resolutions](https://mlsanity.com/paper/2503.05600/) (2025)
- [Stereo World Model: Camera-Guided Stereo Video Generation](https://mlsanity.com/paper/2603.17375/) (2026)
- [4D-MoDe: Towards Editable and Scalable Volumetric Streaming via Motion-Decoupled 4D Gaussian Compression](https://mlsanity.com/paper/2509.17506/) (2025)
- [Vectorized Video Representation with Easy Editing via Hierarchical Spatio-Temporally Consistent Proxy Embedding](https://mlsanity.com/paper/2510.12256/) (2025)
- [Mono4DGS-HDR: High Dynamic Range 4D Gaussian Splatting from Alternating-exposure Monocular Videos](https://mlsanity.com/paper/2510.18489/) (2025)
- [Splat-SAP: Feed-Forward Gaussian Splatting for Human-Centered Scene with Scale-Aware Point Map Reconstruction](https://mlsanity.com/paper/2511.22704/) (2025)
- [Contour Information Aware 2D Gaussian Splatting for Image Representation](https://mlsanity.com/paper/2512.23255/) (2025)
- [Rasterizing Wireless Radiance Field via Deformable 2D Gaussian Splatting](https://mlsanity.com/paper/2506.12787/) (2025)
- [Versatile Video Representation via Feed-Forward 2D Gaussian Splatting Tokenization](https://mlsanity.com/paper/2508.11183/) (2025)
- [Mango-GS: Enhancing Spatio-Temporal Consistency in Dynamic Scenes Reconstruction using Multi-Frame Node-Guided 4D Gaussian Splatting](https://mlsanity.com/paper/2603.11543/) (2026)
- [ReFlow: Self-correction Motion Learning for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2604.01561/) (2026)
- [4D Reconstruction from Sparse Dynamic Cameras](https://mlsanity.com/paper/2606.04593/) (2026)
- [HyperGS: Fast and Generalizable Gaussian Video Representation](https://mlsanity.com/paper/2607.11500/) (2026)
- [TOM-GS: Editable Video Representation via Temporal Opacity Modulation of Static 3D Gaussians](https://mlsanity.com/paper/2607.22717/) (2026)
- [Monocular Depth Estimation from a Single Image: Progress and Opportunities](https://mlsanity.com/paper/2609.01172/) (2026)

## Abstract (arXiv)

Video representation is a long-standing problem that is crucial for various down-stream tasks, such as tracking,depth prediction,segmentation,view synthesis,and editing. However, current methods either struggle to model complex motions due to the absence of 3D structure or rely on implicit 3D representations that are ill-suited for manipulation tasks. To address these challenges, we introduce a novel explicit 3D representation-video Gaussian representation -- that embeds a video into 3D Gaussians. Our proposed representation models video appearance in a 3D canonical space using explicit Gaussians as proxies and associates each Gaussian with 3D motions for video motion. This approach offers a more intrinsic and explicit representation than layered atlas or volumetric pixel matrices. To obtain such a representation, we distill 2D priors, such as optical flow and depth, from foundation models to regularize learning in this ill-posed setting. Extensive applications demonstrate the versatility of our new video representation. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation. Project page: https://sunyangtian.github.io/spatter_a_video_web/