# MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis arXiv: https://arxiv.org/abs/2311.02778 ยท HTML: https://mlsanity.com/paper/2311.02778/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 48 references that are in the map. - First posted on arXiv 5 Nov 2023. ## Builds on (most cited first, 27 of 27) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021) - [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021) - [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021) - [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017) - [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021) - [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019) - [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020) - [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021) - [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018) - [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023) - [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023) - [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021) - [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022) - [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021) - [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021) - [BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis](https://mlsanity.com/paper/2302.14859/) (2023) - [NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM](https://mlsanity.com/paper/2302.03594/) (2023) - [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022) - [BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration](https://mlsanity.com/paper/1604.01093/) (2016) - [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023) - [FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality](https://mlsanity.com/paper/2103.16365/) (2021) - [GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction](https://mlsanity.com/paper/2206.14735/) (2022) - [NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes](https://mlsanity.com/paper/2303.09431/) (2023) - [Boosting Point Clouds Rendering via Radiance Mapping](https://mlsanity.com/paper/2210.15107/) (2022) - [ResNeRF: Geometry-Guided Residual Neural Radiance Field for Indoor Scene Novel View Synthesis](https://mlsanity.com/paper/2211.16211/) (2022) ## Built on by (most cited first, 12 of 12) - [DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing](https://mlsanity.com/paper/2403.17822/) (2024) - [Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation](https://mlsanity.com/paper/2412.14015/) (2024) - [TransAnimate: Taming Layer Diffusion to Generate RGBA Video](https://mlsanity.com/paper/2503.17934/) (2025) - [EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device](https://mlsanity.com/paper/2509.17430/) (2025) - [AGS-Mesh: Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones](https://mlsanity.com/paper/2411.19271/) (2024) - [GLS: Geometry-aware 3D Language Gaussian Splatting](https://mlsanity.com/paper/2411.18066/) (2024) - [FIORD: A Fisheye Indoor-Outdoor Dataset with LIDAR Ground Truth for 3D Scene Reconstruction and Benchmarking](https://mlsanity.com/paper/2504.01732/) (2025) - [GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Models](https://mlsanity.com/paper/2406.12671/) (2024) - [Point Cloud Self-supervised Learning via 3D to Multi-view Masked Learner](https://mlsanity.com/paper/2311.10887/) (2023) - [PlanarGS: High-Fidelity Indoor 3D Gaussian Splatting Guided by Vision-Language Planar Priors](https://mlsanity.com/paper/2510.23930/) (2025) - [OMeGa: Joint Optimization of Explicit Meshes and Gaussian Splats for Robust Scene-Level Surface Reconstruction](https://mlsanity.com/paper/2509.24308/) (2025) - [ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation](https://mlsanity.com/paper/2609.23910/) (2026) ## Abstract (arXiv) Metaverse technologies demand accurate, real-time, and immersive modeling on consumer-grade hardware for both non-human perception (e.g., drone/robot/autonomous car navigation) and immersive technologies like AR/VR, requiring both structural accuracy and photorealism. However, there exists a knowledge gap in how to apply geometric reconstruction and photorealism modeling (novel view synthesis) in a unified framework. To address this gap and promote the development of robust and immersive modeling and rendering with consumer-grade devices, we propose a real-world Multi-Sensor Hybrid Room Dataset (MuSHRoom). Our dataset presents exciting challenges and requires state-of-the-art methods to be cost-effective, robust to noisy data and devices, and can jointly learn 3D reconstruction and novel view synthesis instead of treating them as separate tasks, making them ideal for real-world applications. We benchmark several famous pipelines on our dataset for joint 3D mesh reconstruction and novel view synthesis. Our dataset and benchmark show great potential in promoting the improvements for fusing 3D reconstruction and high-quality rendering in a robust and computationally efficient end-to-end fashion. The dataset and code are available at the project website: https://xuqianren.github.io/publications/MuSHRoom/.