# RelayGS: Reconstructing Dynamic Scenes with Large-Scale and Complex Motions via Relay Gaussians

arXiv: https://arxiv.org/abs/2412.02493 · HTML: https://mlsanity.com/paper/2412.02493/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction](https://mlsanity.com/paper/2403.11447/) (2024)
- [GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis](https://mlsanity.com/paper/2312.11458/) (2023)
- [SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction](https://mlsanity.com/paper/2409.11211/) (2024)
- [MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors](https://mlsanity.com/paper/2406.00434/) (2024)
- [Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction](https://mlsanity.com/paper/2406.03697/) (2024)
- [EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting](https://mlsanity.com/paper/2406.19811/) (2024)
- [D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup](https://mlsanity.com/paper/2405.14276/) (2024)
- [H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2408.13036/) (2024)
- [Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting](https://mlsanity.com/paper/2406.01042/) (2024)

## Built on by (most cited first, 3 of 3)

- [MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2501.03714/) (2025)
- [Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction](https://mlsanity.com/paper/2604.23803/) (2026)
- [PD-4DGS:Progressive Decomposition of 4D Gaussian Splatting for Bandwidth-Adaptive Dynamic Scene Streaming](https://mlsanity.com/paper/2605.11427/) (2026)

## Abstract (arXiv)

Reconstructing dynamic scenes with large-scale and complex motions remains a significant challenge. Recent techniques like Neural Radiance Fields and 3D Gaussian Splatting (3DGS) have shown promise but still struggle with scenes involving substantial movement. This paper proposes RelayGS, a novel method based on 3DGS, specifically designed to represent and reconstruct highly dynamic scenes. Our RelayGS learns a complete 4D representation with canonical 3D Gaussians and a compact motion field, consisting of three stages. First, we learn a fundamental 3DGS from all frames, ignoring temporal scene variations, and use a learnable mask to separate the highly dynamic foreground from the minimally moving background. Second, we replicate multiple copies of the decoupled foreground Gaussians from the first stage, each corresponding to a temporal segment, and optimize them using pseudo-views constructed from multiple frames within each segment. These Gaussians, termed Relay Gaussians, act as explicit relay nodes, simplifying and breaking down large-scale motion trajectories into smaller, manageable segments. Finally, we jointly learn the scene's temporal motion and refine the canonical Gaussians learned from the first two stages. We conduct thorough experiments on two dynamic scene datasets featuring large and complex motions, where our RelayGS outperforms state-of-the-arts by more than 1 dB in PSNR, and successfully reconstructs real-world basketball game scenes in a much more complete and coherent manner, whereas previous methods usually struggle to capture the complex motion of players. Code will be publicly available at https://github.com/gqk/RelayGS