# HyperGS: Fast and Generalizable Gaussian Video Representation

arXiv: https://arxiv.org/abs/2607.11500 · HTML: https://mlsanity.com/paper/2607.11500/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 13 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [HNeRV: A Hybrid Neural Representation for Videos](https://mlsanity.com/paper/2304.02633/) (2023)
- [GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting](https://mlsanity.com/paper/2403.08551/) (2024)
- [Transformers as Meta-Learners for Implicit Neural Representations](https://mlsanity.com/paper/2208.02801/) (2022)
- [E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context](https://mlsanity.com/paper/2207.08132/) (2022)
- [Generalizable Implicit Neural Representations via Instance Pattern Composers](https://mlsanity.com/paper/2211.13223/) (2022)
- [Image-GS: Content-Adaptive Image Representation via 2D Gaussians](https://mlsanity.com/paper/2407.01866/) (2024)
- [Splatter a Video: Video Gaussian Representation for Versatile Processing](https://mlsanity.com/paper/2406.13870/) (2024)
- [Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting](https://mlsanity.com/paper/2406.11672/) (2024)
- [GaussianVideo: Efficient Video Representation via Hierarchical Gaussian Splatting](https://mlsanity.com/paper/2501.04782/) (2025)
- [Fast Encoding and Decoding for Implicit Video Representation](https://mlsanity.com/paper/2409.19429/) (2024)
- [STGV: Spatio-Temporal Hash Encoding for Gaussian-based Video Representation](https://mlsanity.com/paper/2604.10910/) (2026)
- [TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos](https://mlsanity.com/paper/2602.16711/) (2026)

## Abstract (arXiv)

Gaussian Splatting has emerged as an effective representation for video, but existing methods rely on per-video optimization. This leads to slow encoding and limits generalization across videos. To amortize this optimization, we propose HyperGS, a feedforward, optimization-free approach that directly predicts Gaussian representations from any video in a single forward pass, speeding up encoding and decoding by orders of magnitude while generalizing to out-of-distribution videos at higher resolutions. In HyperGS, we design a factorized spatiotemporal Transformer to extract tokens from video, and a learnable query-based Transformer to obtain 8-parameter Gaussian representations for each video frame. We find that naively predicting Gaussians across diverse videos induces a needle-like degeneration that collapses training, and address this with a rank-based geometric regularizer whose strength adapts dynamically to stabilize optimization. HyperGS achieves encoding at 104--105× the speed of per-video Gaussian optimization at matched reconstruction quality while generalizing zero-shot to 720p video, enabling higher-resolution rendering without re-encoding. HyperGS improves PSNR by +2.9--3.1 dB over the prior video encoders on K400, SSv2, and UCF101 at a smaller video representation size. By predicting explicit 2D Gaussians in a single forward pass, HyperGS combines the fast, flexible rendering of Gaussian Splatting with the speed and generalization of feedforward prediction, advancing Gaussians as a practical direction for fast and generalizable video representation.