# Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video

arXiv: https://arxiv.org/abs/2601.05251 · HTML: https://mlsanity.com/paper/2601.05251/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 75 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)

## Built on by (most cited first, 12 of 12)

- [One Video, One World: Turning Monocular Video into Physical 4D Scenes](https://mlsanity.com/paper/2606.31388/) (2026)
- [Syn4D: A Multiview Synthetic 4D Dataset](https://mlsanity.com/paper/2605.05207/) (2026)
- [AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation](https://mlsanity.com/paper/2603.25726/) (2026)
- [Fast 4D Mesh Generation by Spatio-Temporal Attention Chains](https://mlsanity.com/paper/2605.19786/) (2026)
- [Helix4D: Complex 4D Mesh Generation](https://mlsanity.com/paper/2605.26109/) (2026)
- [Follow Your Track: Precise Skeleton Animation Controlled by 3D Trajectories](https://mlsanity.com/paper/2606.25344/) (2026)
- [R-DMesh: Video-Guided 3D Animation via Rectified Dynamic Mesh Flow](https://mlsanity.com/paper/2605.13838/) (2026)
- [MORPHOS: Autoregressive 4D Generation with Temporal Structured Latents](https://mlsanity.com/paper/2606.02491/) (2026)
- [Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text](https://mlsanity.com/paper/2606.05162/) (2026)
- [DynaTok: Token-Based 4D Reconstruction from Partial Point Clouds](https://mlsanity.com/paper/2606.12189/) (2026)
- [RegHead: Non-Humanoid Head Blendshapes via Feed-Forward Registration](https://mlsanity.com/paper/2607.12206/) (2026)
- [BLARM: Animating 3D Objects from Video via Blending Latent Rigid Motion Primitives](https://mlsanity.com/paper/2608.31113/) (2026)

## Abstract (arXiv)

We propose Mesh4D, a feed-forward model for monocular 4D mesh reconstruction. Given a monocular video of a dynamic object, our model reconstructs the object's complete 3D shape and motion, represented as a deformation field. Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass. This latent space is learned by an autoencoder that, during training, is guided by the skeletal structure of the training objects, providing strong priors on plausible deformations. Crucially, skeletal information is not required at inference time. The encoder employs spatio-temporal attention, yielding a more stable representation of the object's overall deformation. Building on this representation, we train a latent diffusion model that, conditioned on the input video and the mesh reconstructed from the first frame, predicts the full animation in one shot. We evaluate Mesh4D on reconstruction and novel view synthesis benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.