# Single Image 3D Interpreter Network

arXiv: https://arxiv.org/abs/1604.08685 · HTML: https://mlsanity.com/paper/1604.08685/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2016.

## Built on by (most cited first, 19 of 19)

- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [Pose for Everything: Towards Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2207.10387/) (2022)
- [ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Poses](https://mlsanity.com/paper/2112.07088/) (2021)
- [KAMA: 3D Keypoint Aware Body Mesh Articulation](https://mlsanity.com/paper/2104.13502/) (2021)
- [PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision](https://mlsanity.com/paper/2203.15625/) (2022)
- [X-Pose: Detecting Any Keypoints](https://mlsanity.com/paper/2310.08530/) (2023)
- [FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction](https://mlsanity.com/paper/2205.07763/) (2022)
- [Self-Supervised 3D Human Pose Estimation with Multiple-View Geometry](https://mlsanity.com/paper/2108.07777/) (2021)
- [CapeX: Category-Agnostic Pose Estimation from Textual Point Explanation](https://mlsanity.com/paper/2406.00384/) (2024)
- [Learning Temporal 3D Human Pose Estimation with Pseudo-Labels](https://mlsanity.com/paper/2110.07578/) (2021)
- [HM3D-ABO: A Photo-realistic Dataset for Object-centric Multi-view 3D Reconstruction](https://mlsanity.com/paper/2206.12356/) (2022)
- [Novel View Synthesis from a Single RGBD Image for Indoor Scenes](https://mlsanity.com/paper/2311.01065/) (2023)
- [SVMAC: Unsupervised 3D Human Pose Estimation from a Single Image with Single-view-multi-angle Consistency](https://mlsanity.com/paper/2106.05616/) (2021)
- [Single Image 3D Object Estimation with Primitive Graph Networks](https://mlsanity.com/paper/2109.04153/) (2021)
- [Learning Compositional Shape Priors for Few-Shot 3D Reconstruction](https://mlsanity.com/paper/2106.06440/) (2021)
- [Dominant motion identification of multi-particle system using deep learning from video](https://mlsanity.com/paper/2104.12722/) (2021)
- [3D Human Pose Estimation Based on 2D-3D Consistency with Synchronized Adversarial Training](https://mlsanity.com/paper/2106.04274/) (2021)
- [Radar2Shape: 3D Shape Reconstruction from High-Frequency Radar using Multiresolution Signed Distance Functions](https://mlsanity.com/paper/2511.17484/) (2025)
- [GKDT: General Keypoint Detection Transformer](https://mlsanity.com/paper/2607.00752/) (2026)

## Abstract (arXiv)

Understanding 3D object structure from a single image is an important but difficult task in computer vision, mostly due to the lack of 3D object annotations in real images. Previous work tackles this problem by either solving an optimization task given 2D keypoint positions, or training on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data. This is made possible mainly by two technical innovations. First, we propose a Projection Layer, which projects estimated 3D structure to 2D space, so that 3D-INN can be trained to predict 3D structural parameters supervised by 2D annotations on real images. Second, heatmaps of keypoints serve as an intermediate representation connecting real and synthetic data, enabling 3D-INN to benefit from the variation and abundance of synthetic 3D objects, without suffering from the difference between the statistics of real and synthesized images due to imperfect rendering. The network achieves state-of-the-art performance on both 2D keypoint estimation and 3D structure recovery. We also show that the recovered 3D information can be used in other vision applications, such as 3D rendering and image retrieval.