# A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing

arXiv: https://arxiv.org/abs/2207.10614 · HTML: https://mlsanity.com/paper/2207.10614/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset](https://mlsanity.com/paper/2004.12276/) (2020)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [Deep Texture Manifold for Ground Terrain Recognition](https://mlsanity.com/paper/1803.10896/) (2018)
- [A 4D Light-Field Dataset and CNN Architectures for Material Recognition](https://mlsanity.com/paper/1608.06985/) (2016)
- [Deep Learning for Tactile Understanding From Visual and Haptic Data](https://mlsanity.com/paper/1511.06065/) (2015)
- [Recognizing Material Properties from Images](https://mlsanity.com/paper/1801.03127/) (2018)
- [Differential Angular Imaging for Material Recognition](https://mlsanity.com/paper/1612.02372/) (2016)
- [A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition](https://mlsanity.com/paper/1703.04699/) (2017)
- [Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision](https://mlsanity.com/paper/2012.02996/) (2020)
- [Material Recognition CNNs and Hierarchical Planning for Biped Robot Locomotion on Slippery Terrain](https://mlsanity.com/paper/1706.08685/) (2017)

## Built on by (most cited first, 23 of 23)

- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](https://mlsanity.com/paper/2404.17569/) (2024)
- [Materialistic: Selecting Similar Materials in Images](https://mlsanity.com/paper/2305.13291/) (2023)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [ZeST: Zero-Shot Material Transfer from a Single Image](https://mlsanity.com/paper/2404.06425/) (2024)
- [MMSFormer: Multimodal Transformer for Material and Semantic Segmentation](https://mlsanity.com/paper/2309.04001/) (2023)
- [SAMa: Material-aware 3D Selection and Segmentation](https://mlsanity.com/paper/2411.19322/) (2024)
- [MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments](https://mlsanity.com/paper/2302.14595/) (2023)
- [MARBLE: Material Recomposition and Blending in CLIP-Space](https://mlsanity.com/paper/2506.05313/) (2025)
- [MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets](https://mlsanity.com/paper/2404.13923/) (2024)
- [One-shot recognition of any material anywhere using contrastive learning with physics-based rendering](https://mlsanity.com/paper/2212.00648/) (2022)
- [Shape and Texture Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2503.23062/) (2025)
- [Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data](https://mlsanity.com/paper/2403.03309/) (2024)
- [LatentBKI: Open-Dictionary Continuous Mapping in Visual-Language Latent Spaces with Quantifiable Uncertainty](https://mlsanity.com/paper/2410.11783/) (2024)
- [Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions](https://mlsanity.com/paper/2604.11579/) (2026)
- [Sonify Anything: Towards Context-Aware Sonic Interactions in AR](https://mlsanity.com/paper/2508.01789/) (2025)
- [With Great Context Comes Great Prediction Power: Classifying Objects via Geo-Semantic Scene Graphs](https://mlsanity.com/paper/2512.23024/) (2025)
- [FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification](https://mlsanity.com/paper/2603.17390/) (2026)
- [Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox](https://mlsanity.com/paper/2605.23747/) (2026)
- [Sub-Semantic Image Segmentation](https://mlsanity.com/paper/2606.14754/) (2026)
- [Unified Video Dense Prediction from Disjoint Data](https://mlsanity.com/paper/2607.21592/) (2026)
- [BIMScript: Material-Aware Structured Scene Programs for BIM Ingestion](https://mlsanity.com/paper/2608.21447/) (2026)
- [Spectral World Renderer: Unified 3D Hyperspectral Unmixing](https://mlsanity.com/paper/2610.19520/) (2026)

## Abstract (arXiv)

A key algorithm for understanding the world is material segmentation, which assigns a label (metal, glass, etc.) to each pixel. We find that a model trained on existing data underperforms in some settings and propose to address this with a large-scale dataset of 3.2 million dense segments on 44,560 indoor and outdoor images, which is 23x more segments than existing data. Our data covers a more diverse set of scenes, objects, viewpoints and materials, and contains a more fair distribution of skin types. We show that a model trained on our data outperforms a state-of-the-art model across datasets and viewpoints. We propose a large-scale scene parsing benchmark and baseline of 0.729 per-pixel accuracy, 0.585 mean class accuracy and 0.420 mean IoU across 46 materials.