# Approaching human 3D shape perception with neurally mappable models

arXiv: https://arxiv.org/abs/2308.11300 · HTML: https://mlsanity.com/paper/2308.11300/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 72 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [The neuroconnectionist research programme](https://mlsanity.com/paper/2209.03718/) (2022)
- [Visual Affordance and Function Understanding: A Survey](https://mlsanity.com/paper/1807.06775/) (2018)
- [Progress and limitations of deep networks to recognize objects in unusual poses](https://mlsanity.com/paper/2207.08034/) (2022)
- [Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training](https://mlsanity.com/paper/2110.01476/) (2021)
- [Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations](https://mlsanity.com/paper/2109.13445/) (2021)

## Built on by (most cited first, 6 of 6)

- [DepthCues: Evaluating Monocular Depth Perception in Large Vision Models](https://mlsanity.com/paper/2411.17385/) (2024)
- [Brain Decodes Deep Nets](https://mlsanity.com/paper/2312.01280/) (2023)
- [Achieving More Human Brain-Like Vision via Human EEG Representational Alignment](https://mlsanity.com/paper/2401.17231/) (2024)
- [Do large language vision models understand 3D shapes?](https://mlsanity.com/paper/2412.10908/) (2024)
- [Human-level 3D shape perception emerges from multi-view learning](https://mlsanity.com/paper/2602.17650/) (2026)
- [QualiaNet: An Experience-Before-Inference Network](https://mlsanity.com/paper/2604.14193/) (2026)

## Abstract (arXiv)

Humans effortlessly infer the 3D shape of objects. What computations underlie this ability? Although various computational models have been proposed, none of them capture the human ability to match object shape across viewpoints. Here, we ask whether and how this gap might be closed. We begin with a relatively novel class of computational models, 3D neural fields, which encapsulate the basic principles of classic analysis-by-synthesis in a deep neural network (DNN). First, we find that a 3D Light Field Network (3D-LFN) supports 3D matching judgments well aligned to humans for within-category comparisons, adversarially-defined comparisons that accentuate the 3D failure cases of standard DNN models, and adversarially-defined comparisons for algorithmically generated shapes with no category structure. We then investigate the source of the 3D-LFN's ability to achieve human-aligned performance through a series of computational experiments. Exposure to multiple viewpoints of objects during training and a multi-view learning objective are the primary factors behind model-human alignment; even conventional DNN architectures come much closer to human behavior when trained with multi-view objectives. Finally, we find that while the models trained with multi-view learning objectives are able to partially generalize to new object categories, they fall short of human alignment. This work provides a foundation for understanding human shape inferences within neurally mappable computational architectures.