# PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation

arXiv: https://arxiv.org/abs/2401.02281 · HTML: https://mlsanity.com/paper/2401.02281/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Point-Based Neural Rendering with Per-View Optimization](https://mlsanity.com/paper/2109.02369/) (2021)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)](https://mlsanity.com/paper/2210.00379/) (2022)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark](https://mlsanity.com/paper/2203.05701/) (2022)
- [Neural Point Catacaustics for Novel-View Synthesis of Reflections](https://mlsanity.com/paper/2301.01087/) (2023)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)
- [On Object Symmetries and 6D Pose Estimation from Images](https://mlsanity.com/paper/1908.07640/) (2019)
- [CherryPicker: Semantic Skeletonization and Topological Reconstruction of Cherry Trees](https://mlsanity.com/paper/2304.04708/) (2023)
- [VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering](https://mlsanity.com/paper/2311.04634/) (2023)
- [NeuralLabeling: A versatile toolset for labeling vision datasets using Neural Radiance Fields](https://mlsanity.com/paper/2309.11966/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination](https://mlsanity.com/paper/2412.14957/) (2024)
- [Interactive World Simulator for Robot Policy Training and Evaluation](https://mlsanity.com/paper/2603.08546/) (2026)
- [VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points](https://mlsanity.com/paper/2410.17932/) (2024)
- [DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction](https://mlsanity.com/paper/2503.05484/) (2025)
- [Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks](https://mlsanity.com/paper/2406.15149/) (2024)
- [Mode-GS: Monocular Depth Guided Anchored 3D Gaussian Splatting for Robust Ground-View Scene Rendering](https://mlsanity.com/paper/2410.04646/) (2024)
- [PMGS: Reconstruction of Projectile Motion Across Large Spatiotemporal Spans via 3D Gaussian Splatting](https://mlsanity.com/paper/2508.02660/) (2025)
- [Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation](https://mlsanity.com/paper/2504.08473/) (2025)
- [Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning](https://mlsanity.com/paper/2505.14938/) (2025)
- [iMaC: Translating Actions into Motion and Contact Images for Embodied World Models](https://mlsanity.com/paper/2606.09813/) (2026)
- [UAVTwin: Neural Digital Twins for UAVs using Gaussian Splatting](https://mlsanity.com/paper/2504.02158/) (2025)
- [Computer vision training dataset generation for robotic environments using Gaussian splatting](https://mlsanity.com/paper/2512.13411/) (2025)

## Abstract (arXiv)

We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. <i>PEGASUS</i> allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.