# Sketch-guided Cage-based 3D Gaussian Splatting Deformation

arXiv: https://arxiv.org/abs/2411.12168 · HTML: https://mlsanity.com/paper/2411.12168/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models](https://mlsanity.com/paper/2312.13763/) (2023)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [DreamEditor: Text-Driven 3D Scene Editing with Neural Fields](https://mlsanity.com/paper/2306.13455/) (2023)
- [GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing](https://mlsanity.com/paper/2403.08733/) (2024)
- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [SKED: Sketch-guided Text-based 3D Editing](https://mlsanity.com/paper/2303.10735/) (2023)
- [EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion](https://mlsanity.com/paper/2312.06725/) (2023)
- [Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches](https://mlsanity.com/paper/2104.00482/) (2021)
- [Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes](https://mlsanity.com/paper/2205.02904/) (2022)
- [Blending-NeRF: Text-Driven Localized Editing in Neural Radiance Fields](https://mlsanity.com/paper/2308.11974/) (2023)
- [Mesh-based Gaussian Splatting for Real-time Large-scale Deformation](https://mlsanity.com/paper/2402.04796/) (2024)
- [As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors](https://mlsanity.com/paper/2311.16739/) (2023)
- [SketchDream: Sketch-based Text-to-3D Generation and Editing](https://mlsanity.com/paper/2405.06461/) (2024)
- [SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling](https://mlsanity.com/paper/2306.06088/) (2023)
- [Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models](https://mlsanity.com/paper/2312.10835/) (2023)

## Built on by (most cited first, 3 of 3)

- [SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Training](https://mlsanity.com/paper/2512.05354/) (2025)
- [CAGE-GS: High-fidelity Cage Based 3D Gaussian Splatting Deformation](https://mlsanity.com/paper/2504.12800/) (2025)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)

## Abstract (arXiv)

3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.