# Direct and Adaptable Mesh-Gaussian Scene Reconstruction from Multi-View Images

arXiv: https://arxiv.org/abs/2405.06945 · HTML: https://mlsanity.com/paper/2405.06945/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 11 May 2024.

## Builds on (most cited first, 26 of 26)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022)
- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes](https://mlsanity.com/paper/2404.10772/) (2024)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis](https://mlsanity.com/paper/2302.14859/) (2023)
- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [Flexible Isosurface Extraction for Gradient-Based Mesh Optimization](https://mlsanity.com/paper/2308.05371/) (2023)
- [NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance](https://mlsanity.com/paper/2312.00846/) (2023)
- [Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement](https://mlsanity.com/paper/2303.02091/) (2023)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting](https://mlsanity.com/paper/2402.01459/) (2024)
- [Robust Watertight Manifold Surface Generation Method for ShapeNet Models](https://mlsanity.com/paper/1802.01698/) (2018)
- [NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes](https://mlsanity.com/paper/2303.09431/) (2023)
- [Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Rendering](https://mlsanity.com/paper/2401.15318/) (2024)

## Built on by (most cited first, 9 of 9)

- [GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering](https://mlsanity.com/paper/2410.24204/) (2024)
- [GauSTAR: Gaussian Surface Tracking and Reconstruction](https://mlsanity.com/paper/2501.10283/) (2025)
- [VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians](https://mlsanity.com/paper/2506.02741/) (2025)
- [Radiance Meshes for Volumetric Reconstruction](https://mlsanity.com/paper/2512.04076/) (2025)
- [Creating Your Editable 3D Photorealistic Avatar with Tetrahedron-constrained Gaussian Splatting](https://mlsanity.com/paper/2504.20403/) (2025)
- [SGAD-SLAM: Splatting Gaussians at Adjusted Depth for Better Radiance Fields in RGBD SLAM](https://mlsanity.com/paper/2603.21055/) (2026)
- [OMeGa: Joint Optimization of Explicit Meshes and Gaussian Splats for Robust Scene-Level Surface Reconstruction](https://mlsanity.com/paper/2509.24308/) (2025)
- [Gaussian Pixel Codec Avatars: A Hybrid Representation for Efficient Rendering](https://mlsanity.com/paper/2512.15711/) (2025)
- [Gaussian Mesh Renderer for Lightweight Differentiable Rendering](https://mlsanity.com/paper/2602.14493/) (2026)

## Abstract (arXiv)

Jointly recovering explicit surface geometry and high-quality appearance from multi-view images remains challenging. This capability is essential for maintaining high-fidelity real-to-sim environments for embodied intelligence, where local changes should be incorporated without complete reconstruction. Existing neural surface reconstruction and 3DGS-to-mesh pipelines often learn geometry indirectly or separate geometry construction from appearance modeling. This separation introduces optimization redundancy and makes local geometry or appearance updates expensive. We propose an end-to-end mesh-Gaussian scene representation that binds 3D Gaussians to mesh faces and uses differentiable 3DGS rendering for photometric supervision. This design provides a direct information pathway for jointly learning explicit geometry and renderable appearance. Experiments on indoor and outdoor scenes demonstrate improved efficiency and rendering quality while preserving high-quality surface reconstruction. The explicit mesh also enables mesh-based manipulation, and the coupled representation adapts efficiently to local scene modifications. These properties support scalable visual scene modeling and the efficient maintenance of real-to-sim environments for embodied-agent training and evaluation.