# MagicClay: Sculpting Meshes With Generative Neural Fields

arXiv: https://arxiv.org/abs/2403.02460 · HTML: https://mlsanity.com/paper/2403.02460/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2024.

## Built on by (most cited first, 13 of 13)

- [Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects](https://mlsanity.com/paper/2412.00518/) (2024)
- [Garment3DGen: 3D Garment Stylization and Texture Generation](https://mlsanity.com/paper/2403.18816/) (2024)
- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [Geometry in Style: 3D Stylization via Surface Normal Deformation](https://mlsanity.com/paper/2503.23241/) (2025)
- [CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generation](https://mlsanity.com/paper/2505.07003/) (2025)
- [AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flows](https://mlsanity.com/paper/2511.22357/) (2025)
- [CraftMesh: High-Fidelity Generative Mesh Manipulation via Poisson Seamless Fusion](https://mlsanity.com/paper/2509.13688/) (2025)
- [SuperCarver: Texture-Consistent 3D Geometry Super-Resolution for High-Fidelity Surface Detail Generation](https://mlsanity.com/paper/2503.09439/) (2025)
- [Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting](https://mlsanity.com/paper/2502.20045/) (2025)
- [Handle-based Mesh Deformation Guided By Vision Language Model](https://mlsanity.com/paper/2506.04562/) (2025)
- [ES3D: Embedding Semantics into 3D Space for Component-Aware Editing](https://mlsanity.com/paper/2608.15749/) (2026)
- [RADmesh: Remesh-Aware Mesh Deformation](https://mlsanity.com/paper/2608.17182/) (2026)

## Abstract (arXiv)

The recent developments in neural fields have brought phenomenal capabilities to the field of shape generation, but they lack crucial properties, such as incremental control - a fundamental requirement for artistic work. Triangular meshes, on the other hand, are the representation of choice for most geometry related tasks, offering efficiency and intuitive control, but do not lend themselves to neural optimization. To support downstream tasks, previous art typically proposes a two-step approach, where first a shape is generated using neural fields, and then a mesh is extracted for further processing. Instead, in this paper we introduce a hybrid approach that maintains both a mesh and a Signed Distance Field (SDF) representations consistently. Using this representation, we introduce MagicClay - an artist friendly tool for sculpting regions of a mesh according to textual prompts while keeping other regions untouched. Our framework carefully and efficiently balances consistency between the representations and regularizations in every step of the shape optimization; Relying on the mesh representation, we show how to render the SDF at higher resolutions and faster. In addition, we employ recent work in differentiable mesh reconstruction to adaptively allocate triangles in the mesh where required, as indicated by the SDF. Using an implemented prototype, we demonstrate superior generated geometry compared to the state-of-the-art, and novel consistent control, allowing sequential prompt-based edits to the same mesh for the first time.