mlsanityData to 7 Oct 2026

SAM 3D: 3Dfy Anything in Images

arXiv 2511.16624 ↗First posted 20 Nov 2025cs.CV cs.AIcited by 230

Not on the map. Why?

Abstract (arXiv)

We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image. SAM 3D excels in natural images, where occlusion and scene clutter are common and visual recognition cues from context play a larger role. We achieve this with a human- and model-in-the-loop pipeline for annotating object shape, texture, and pose, providing visually grounded 3D reconstruction data at unprecedented scale. We learn from this data in a modern, multi-stage training framework that combines synthetic pretraining with real-world alignment, breaking the 3D "data barrier". We obtain significant gains over recent work, with at least a 5:1 win rate in human preference tests on real-world objects and scenes. We will release our code and model weights, an online demo, and a new challenging benchmark for in-the-wild 3D object reconstruction.

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). Builds on 93 of its 126 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

93 of 126 references.

References · 93

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Segment Anythingcited by 8,158
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  11. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
38 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  6. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  7. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  8. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  9. 2020Neural Sparse Voxel Fieldscited by 959
  10. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  11. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  12. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  13. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  14. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  15. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  16. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  17. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  18. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  19. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  20. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  21. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
  22. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  23. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  24. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  25. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  26. 2024One Step Diffusion via Shortcut Modelscited by 373
  27. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  28. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  29. 2024MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervisioncited by 290
  30. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  31. 2021Scaling Laws for Transfercited by 265
  32. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  33. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  34. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  35. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  36. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  37. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  38. 2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 204

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 221.

230 papers.

Sort:

By area:

  1. 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 40
  2. 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenescited by 39
  3. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 21
  4. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  5. 2026MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generationcited by 16
  6. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulationcited by 15
  7. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  8. 2026Beyond Pixel Histories: World Models with Persistent 3D Statecited by 11
  9. 2026Pixal3D: Pixel-Aligned 3D Generation from Imagescited by 11
  10. 2026HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videoscited by 10
  11. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  12. 2025SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modelingcited by 9
38 more
  1. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  2. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  3. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 6
  4. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  5. 2026Stream3D: Sequential Multi-View 3D Generation via Evidential Memorycited by 6
  6. 2026REST3D: Reconstructing Physically Stable 3D Scenes from a Single Imagecited by 6
  7. 2026Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Modelscited by 6
  8. 2026Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstrationcited by 6
  9. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agentscited by 6
  10. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  11. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 6
  12. 2026TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulationcited by 5
  13. 2026MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstructioncited by 5
  14. 2026ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstructioncited by 5
  15. 2026MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstructioncited by 5
  16. 2026Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3Dcited by 5
  17. 2026Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observationscited by 5
  18. 2025GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherencecited by 4
  19. 2026ObjectForesight: Predicting Future 3D Object Trajectories from Human Videoscited by 4
  20. 2026AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generationcited by 4
  21. 2026Picasso: Holistic Scene Reconstruction with Physics-Constrained Samplingcited by 4
  22. 2026Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priorscited by 4
  23. 2026SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Imagecited by 4
  24. 2026EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AIcited by 4
  25. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  26. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  27. 2026TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkourcited by 3
  28. 2026SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequencescited by 3
  29. 2026Fast-SAM3D: 3Dfy Anything in Images but Fastercited by 3
  30. 2026Simulation-Ready Cluttered Scene Estimation via Physics-aware Joint Shape and Pose Optimizationcited by 3
  31. 2026SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLMcited by 3
  32. 2026RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Spacecited by 3
  33. 2026neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editingcited by 3
  34. 2026Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimationcited by 3
  35. 2026CRONOS: Benchmarking Counterfactual Physical Consistency in Video Modelscited by 3
  36. 2026World Tracing: Generative Pixel-Aligned Geometry Beyond the Visiblecited by 3
  37. 2026Modality Forcing for Scalable Spatial Generationcited by 3
  38. 2026φ-Scene: Physically Grounded Image-to-3D Scene Reconstructioncited by 3

50 most cited of 230 All 230 (JSON)