# VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations

arXiv: https://arxiv.org/abs/2208.00949 · HTML: https://mlsanity.com/paper/2208.00949/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [NeRF-Editing: Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2205.04978/) (2022)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)

## Built on by (most cited first, 30 of 36)

- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [Drivable 3D Gaussian Avatars](https://mlsanity.com/paper/2311.08581/) (2023)
- [Vox-E: Text-guided Voxel Editing of 3D Objects](https://mlsanity.com/paper/2303.12048/) (2023)
- [Adaptive Shells for Efficient Neural Radiance Field Rendering](https://mlsanity.com/paper/2311.10091/) (2023)
- [Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data](https://mlsanity.com/paper/2311.18729/) (2023)
- [StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video](https://mlsanity.com/paper/2305.00942/) (2023)
- [PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF](https://mlsanity.com/paper/2311.13099/) (2023)
- [Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video](https://mlsanity.com/paper/2211.12782/) (2022)
- [Human Hair Reconstruction with Strand-Aligned 3D Gaussians](https://mlsanity.com/paper/2409.14778/) (2024)
- [CoGS: Controllable Gaussian Splatting](https://mlsanity.com/paper/2312.05664/) (2023)
- [Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation](https://mlsanity.com/paper/2407.09497/) (2024)
- [MEGANE: Morphable Eyeglass and Avatar Network](https://mlsanity.com/paper/2302.04868/) (2023)
- [Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training](https://mlsanity.com/paper/2312.01663/) (2023)
- [DINER: Depth-aware Image-based NEural Radiance fields](https://mlsanity.com/paper/2211.16630/) (2022)
- [VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence](https://mlsanity.com/paper/2405.16204/) (2024)
- [CoARF: Controllable 3D Artistic Style Transfer for Radiance Fields](https://mlsanity.com/paper/2404.14967/) (2024)
- [Synthetic Prior for Few-Shot Drivable Head Avatar Inversion](https://mlsanity.com/paper/2501.06903/) (2025)
- [Neural Impostor: Editing Neural Radiance Fields with Explicit Shape Manipulation](https://mlsanity.com/paper/2310.05391/) (2023)
- [Point-Based Radiance Fields for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2310.03375/) (2023)
- [Spice-E : Structural Priors in 3D Diffusion using Cross-Entity Attention](https://mlsanity.com/paper/2311.17834/) (2023)
- [BlendFields: Few-Shot Example-Driven Facial Modeling](https://mlsanity.com/paper/2305.07514/) (2023)
- [LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control](https://mlsanity.com/paper/2406.16038/) (2024)
- [BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading](https://mlsanity.com/paper/2506.06271/) (2025)
- [Factored Neural Representation for Scene Understanding](https://mlsanity.com/paper/2304.10950/) (2023)
- [Affine-Equivariant Kernel Space Encoding for NeRF Editing](https://mlsanity.com/paper/2508.02831/) (2025)
- [Retargeting Visual Data with Deformation Fields](https://mlsanity.com/paper/2311.13297/) (2023)
- [GSDeformer: Direct, Real-time and Extensible Cage-based Deformation for 3D Gaussian Splatting](https://mlsanity.com/paper/2405.15491/) (2024)
- [VoluMe -- Authentic 3D Video Calls from Live Gaussian Splat Prediction](https://mlsanity.com/paper/2507.21311/) (2025)
- [DualNeRF: Text-Driven 3D Scene Editing via Dual-Field Representation](https://mlsanity.com/paper/2502.16302/) (2025)

## Abstract (arXiv)

The recent increase in popularity of volumetric representations for scene reconstruction and novel view synthesis has put renewed focus on animating volumetric content at high visual quality and in real-time. While implicit deformation methods based on learned functions can produce impressive results, they are `black boxes' to artists and content creators, they require large amounts of training data to generalise meaningfully, and they do not produce realistic extrapolations outside the training data. In this work we solve these issues by introducing a volume deformation method which is real-time, easy to edit with off-the-shelf software and can extrapolate convincingly. To demonstrate the versatility of our method, we apply it in two scenarios: physics-based object deformation and telepresence where avatars are controlled using blendshapes. We also perform thorough experiments showing that our method compares favourably to both volumetric approaches combined with implicit deformation and methods based on mesh deformation.