# Marr Revisited: 2D-3D Alignment via Surface Normal Prediction

arXiv: https://arxiv.org/abs/1604.01347 · HTML: https://mlsanity.com/paper/1604.01347/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2016.

## Built on by (most cited first, 30 of 34)

- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal](https://mlsanity.com/paper/2406.16864/) (2024)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Rethinking Inductive Biases for Surface Normal Estimation](https://mlsanity.com/paper/2403.00712/) (2024)
- [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation](https://mlsanity.com/paper/2109.09881/) (2021)
- [Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think](https://mlsanity.com/paper/2409.11355/) (2024)
- [Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single View](https://mlsanity.com/paper/2404.03421/) (2024)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [Dens3R: A Foundation Model for 3D Geometry Prediction](https://mlsanity.com/paper/2507.16290/) (2025)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [IronDepth: Iterative Refinement of Single-View Depth using Surface Normal and its Uncertainty](https://mlsanity.com/paper/2210.03676/) (2022)
- [SNE-RoadSeg+: Rethinking Depth-Normal Translation and Deep Supervision for Freespace Detection](https://mlsanity.com/paper/2107.14599/) (2021)
- [Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion](https://mlsanity.com/paper/2412.09593/) (2024)
- [Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors](https://mlsanity.com/paper/2411.17249/) (2024)
- [Egocentric Scene Understanding via Multimodal Spatial Rectifier](https://mlsanity.com/paper/2207.07077/) (2022)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [DeepJoin: Learning a Joint Occupancy, Signed Distance, and Normal Field Function for Shape Repair](https://mlsanity.com/paper/2211.12400/) (2022)
- [Modeling 3D Surface Manifolds with a Locally Conditioned Atlas](https://mlsanity.com/paper/2102.05984/) (2021)
- [Towards Panoptic 3D Parsing for Single Image in the Wild](https://mlsanity.com/paper/2111.03039/) (2021)
- [Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database](https://mlsanity.com/paper/2207.01339/) (2022)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [Normal Transformer: Extracting Surface Geometry from LiDAR Points Enhanced by Visual Semantics](https://mlsanity.com/paper/2211.10580/) (2022)
- [MonoPlane: Exploiting Monocular Geometric Cues for Generalizable 3D Plane Reconstruction](https://mlsanity.com/paper/2411.01226/) (2024)
- [Monocular Normal Estimation via Shading Sequence Estimation](https://mlsanity.com/paper/2602.09929/) (2026)
- [Neural Network Normal Estimation and Bathymetry Reconstruction from Sidescan Sonar](https://mlsanity.com/paper/2206.07819/) (2022)
- [Robust and Real-time Surface Normal Estimation from Stereo Disparities using Affine Transformations](https://mlsanity.com/paper/2504.15121/) (2025)
- [Hybrid Approach for 3D Head Reconstruction: Using Neural Networks and Visual Geometry](https://mlsanity.com/paper/2104.13710/) (2021)
- [Face Inverse Rendering via Hierarchical Decoupling](https://mlsanity.com/paper/2301.06733/) (2023)

## Abstract (arXiv)

We introduce an approach that leverages surface normal predictions, along with appearance cues, to retrieve 3D models for objects depicted in 2D still images from a large CAD object library. Critical to the success of our approach is the ability to recover accurate surface normals for objects in the depicted scene. We introduce a skip-network model built on the pre-trained Oxford VGG convolutional neural network (CNN) for surface normal prediction. Our model achieves state-of-the-art accuracy on the NYUv2 RGB-D dataset for surface normal prediction, and recovers fine object detail compared to previous methods. Furthermore, we develop a two-stream network over the input image and predicted surface normals that jointly learns pose and style for CAD model retrieval. When using the predicted surface normals, our two-stream network matches prior work using surface normals computed from RGB-D images on the task of pose prediction, and achieves state of the art when using RGB-D input. Finally, our two-stream network allows us to retrieve CAD models that better match the style and pose of a depicted object compared with baseline approaches.