# Materialistic: Selecting Similar Materials in Images

arXiv: https://arxiv.org/abs/2305.13291 · HTML: https://mlsanity.com/paper/2305.13291/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 94 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 46)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [Deep Polarization Imaging for 3D shape and SVBRDF Acquisition](https://mlsanity.com/paper/2105.02875/) (2021)

## Built on by (most cited first, 26 of 26)

- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [Alchemist: Parametric Control of Material Properties with Diffusion Models](https://mlsanity.com/paper/2312.02970/) (2023)
- [Material Palette: Extraction of Materials from a Single Image](https://mlsanity.com/paper/2311.17060/) (2023)
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](https://mlsanity.com/paper/2404.17569/) (2024)
- [ZeST: Zero-Shot Material Transfer from a Single Image](https://mlsanity.com/paper/2404.06425/) (2024)
- [OmniSR: Shadow Removal under Direct and Indirect Lighting](https://mlsanity.com/paper/2410.01719/) (2024)
- [When Does Perceptual Alignment Benefit Vision Representations?](https://mlsanity.com/paper/2410.10817/) (2024)
- [NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs](https://mlsanity.com/paper/2402.08622/) (2024)
- [SAMa: Material-aware 3D Selection and Segmentation](https://mlsanity.com/paper/2411.19322/) (2024)
- [MARBLE: Material Recomposition and Blending in CLIP-Space](https://mlsanity.com/paper/2506.05313/) (2025)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [ARM: Appearance Reconstruction Model for Relightable 3D Generation](https://mlsanity.com/paper/2411.10825/) (2024)
- [MatSwap: Light-aware material transfers in images](https://mlsanity.com/paper/2502.07784/) (2025)
- [Inferring Dynamic Physical Properties from Video Foundation Models](https://mlsanity.com/paper/2510.02311/) (2025)
- [DenseSR: Image Shadow Removal as Dense Prediction](https://mlsanity.com/paper/2507.16472/) (2025)
- [Fine-Grained Spatially Varying Material Selection in Images](https://mlsanity.com/paper/2506.09023/) (2025)
- [UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields](https://mlsanity.com/paper/2506.21884/) (2025)
- [Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data](https://mlsanity.com/paper/2403.03309/) (2024)
- [MaRI: Material Retrieval Integration across Domains](https://mlsanity.com/paper/2503.08111/) (2025)
- [Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions](https://mlsanity.com/paper/2604.11579/) (2026)
- [SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art](https://mlsanity.com/paper/2511.01817/) (2025)
- [Phieat: Physically Grounded Material Feature Representation](https://mlsanity.com/paper/2511.11270/) (2025)
- [Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes](https://mlsanity.com/paper/2603.17370/) (2026)
- [MAOAM: Unified Object and Material Selection with Vision-Language Models](https://mlsanity.com/paper/2606.04880/) (2026)
- [Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Models](https://mlsanity.com/paper/2606.22945/) (2026)
- [Diffusion-Based Material Regularization for Physics-Based Inverse Rendering](https://mlsanity.com/paper/2606.31065/) (2026)

## Abstract (arXiv)

Separating an image into meaningful underlying components is a crucial first step for both editing and understanding images. We present a method capable of selecting the regions of a photograph exhibiting the same material as an artist-chosen area. Our proposed approach is robust to shading, specular highlights, and cast shadows, enabling selection in real images. As we do not rely on semantic segmentation (different woods or metal should not be selected together), we formulate the problem as a similarity-based grouping problem based on a user-provided image location. In particular, we propose to leverage the unsupervised DINO features coupled with a proposed Cross-Similarity module and an MLP head to extract material similarities in an image. We train our model on a new synthetic image dataset, that we release. We show that our method generalizes well to real-world images. We carefully analyze our model's behavior on varying material properties and lighting. Additionally, we evaluate it against a hand-annotated benchmark of 50 real photographs. We further demonstrate our model on a set of applications, including material editing, in-video selection, and retrieval of object photographs with similar materials.