mlsanityData to 7 Oct 2026

Matrix3D: Large Photogrammetry Model All-in-One

arXiv 2502.07685 ↗First posted 11 Feb 2025cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D's large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation. Project page: https://nju-3dv.github.io/projects/matrix3d.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 112 of its 148 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

112 of 148 references.

References · 112

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  11. 2024Depth Anything V2cited by 1,385
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  13. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  14. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  15. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  16. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  17. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  18. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  19. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  20. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  21. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  22. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  23. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  24. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  25. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  26. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  27. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  28. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  29. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  30. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  31. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  32. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  33. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  34. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  35. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  36. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  37. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  38. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425

50 most cited of 112 All 112 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 23.

39 papers.

Sort:

By area:

  1. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 295
  2. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  3. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
  4. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  5. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  6. 2026LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstructioncited by 32
  7. 2025DiT360: High-Fidelity Panoramic Image Generation via Hybrid Trainingcited by 24
  8. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 24
  9. 2026Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Videocited by 12
  10. 2026Repurposing Geometric Foundation Models for Multi-view Diffusioncited by 12
  11. 20253R-GS: Best Practice in Optimizing Camera Poses Along with 3DGScited by 10
  12. 2025How Much 3D Do Video Foundation Models Encode?cited by 9
27 more
  1. 2025One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Controlcited by 8
  2. 2025CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Modelscited by 8
  3. 2025FastMap: Revisiting Structure from Motion through First-Order Optimizationcited by 7
  4. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 6
  5. 2026GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generatorcited by 6
  6. 2026ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Modelscited by 5
  7. 2025Lightweight and Accurate Multi-View Stereo with Confidence-Aware Diffusion Modelcited by 4
  8. 2025Terra: Explorable Native 3D World Model with Point Latentscited by 4
  9. 2025RayPose: Ray Bundling Diffusion for Template Views in Unseen 6D Object Pose Estimationcited by 4
  10. 2025Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interfacecited by 4
  11. 2026Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectoriescited by 4
  12. 2026MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAEcited by 3
  13. 2026SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenescited by 3
  14. 2026MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffoldcited by 3
  15. 2026RnG: A Unified Transformer for Complete 3D Modeling from Partial Observationscited by 2
  16. 2026UniSHARP: Universal Sharp Monocular View Synthesiscited by 2
  17. 2026HorizonRelight: Relighting Long-horizon Videos Consistently via Diffusion Transformerscited by 1
  18. 2026GeoFF3D: Coordinate-Anchored Feed-Forward Reconstruction for Large-Scale UAV Mappingcited by 1
  19. 2026Gimbal360: Canonicalizing Planar Diffusion for Spherical Panorama Completioncited by 0
  20. 2026Mind the Hitch: Dynamic Calibration and Articulated Perception for Autonomous Truckscited by 0
  21. 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretrainingcited by 0
  22. 2026Advantage-Guided Gate: Reshaping Open-Ended Reasoning for Vision-Based Spatial Intelligencecited by 0
  23. 2026Gaussian Sculpting: End-to-End Controllable Surface Reconstruction via Field Optimizationcited by 0
  24. 2026SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesiscited by 0
  25. 2026NaCR: Visual Localization via NeRF-aided Camera Ray Regressioncited by 0
  26. 2026M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesiscited by 0
  27. 2026GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
  2. 2026Single Image to Textured 3D Object Generation in Frequency Domain: From Theory to Pipeline
  3. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
  4. 2025NOVA3D: Normal Aligned Video Diffusion Model for Single Image to 3D Generation
  5. 2025CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction
  6. 2022MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception
  7. 2025OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformer
  8. 2026CAM3R: Camera-Agnostic Model for 3D Reconstruction