# SplaTraj: Camera Trajectory Generation with Semantic Gaussian Splatting

arXiv: https://arxiv.org/abs/2410.06014 · HTML: https://mlsanity.com/paper/2410.06014/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping](https://mlsanity.com/paper/2403.09637/) (2024)
- [DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark](https://mlsanity.com/paper/2403.10814/) (2024)
- [Geometric Fabrics: Generalizing Classical Mechanics to Capture the Physics of Behavior](https://mlsanity.com/paper/2109.10443/) (2021)
- [Parallelised Diffeomorphic Sampling-based Motion Planning](https://mlsanity.com/paper/2108.11775/) (2021)
- [RecGS: Removing Water Caustic with Recurrent Gaussian Splatting](https://mlsanity.com/paper/2407.10318/) (2024)
- [V-PRISM: Probabilistic Mapping of Unknown Tabletop Scenes](https://mlsanity.com/paper/2403.08106/) (2024)
- [Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction](https://mlsanity.com/paper/1907.05127/) (2019)
- [Directly Optimizing IoU for Bounding Box Localization](https://mlsanity.com/paper/2304.07256/) (2023)

## Built on by (most cited first, 10 of 10)

- [GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photography](https://mlsanity.com/paper/2504.07083/) (2025)
- [SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fields](https://mlsanity.com/paper/2506.09565/) (2025)
- [Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields](https://mlsanity.com/paper/2503.20776/) (2025)
- [ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation](https://mlsanity.com/paper/2603.11421/) (2026)
- [AnimateScene: Camera-controllable Animation in Any Scene](https://mlsanity.com/paper/2508.05982/) (2025)
- [FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views](https://mlsanity.com/paper/2604.09862/) (2026)
- [Infinite Leagues Under the Sea: Photorealistic 3D Underwater Terrain Generation by Latent Fractal Diffusion Models](https://mlsanity.com/paper/2503.06784/) (2025)
- [CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents](https://mlsanity.com/paper/2607.26910/) (2026)
- [CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels](https://mlsanity.com/paper/2608.30342/) (2026)
- [OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning](https://mlsanity.com/paper/2610.09513/) (2026)

## Abstract (arXiv)

Many recent developments for robots to represent environments have focused on photorealistic reconstructions. This paper particularly focuses on generating sequences of images from the photorealistic Gaussian Splatting models, that match instructions that are given by user-inputted language. We contribute a novel framework, SplaTraj, which formulates the generation of images within photorealistic environment representations as a continuous-time trajectory optimization problem. Costs are designed so that a camera following the trajectory poses will smoothly traverse through the environment and render the specified spatial information in a photogenic manner. This is achieved by querying a photorealistic representation with language embedding to isolate regions that correspond to the user-specified inputs. These regions are then projected to the camera's view as it moves over time and a cost is constructed. We can then apply gradient-based optimization and differentiate through the rendering to optimize the trajectory for the defined cost. The resulting trajectory moves to photogenically view each of the specified objects. We empirically evaluate our approach on a suite of environments and instructions, and demonstrate the quality of generated image sequences.