# One-shot recognition of any material anywhere using contrastive learning with physics-based rendering

arXiv: https://arxiv.org/abs/2212.00648 · HTML: https://mlsanity.com/paper/2212.00648/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [Self-Supervised Material and Texture Representation Learning for Remote Sensing Tasks](https://mlsanity.com/paper/2112.01715/) (2021)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing](https://mlsanity.com/paper/2207.10614/) (2022)
- [A Similarity Measure for Material Appearance](https://mlsanity.com/paper/1905.01562/) (2019)
- [Recognizing Material Properties from Images](https://mlsanity.com/paper/1801.03127/) (2018)
- [Sparse Surface Constraints for Combining Physics-based Elasticity Simulation and Correspondence-Free Object Reconstruction](https://mlsanity.com/paper/1910.01812/) (2019)
- [SOLIS: Autonomous Solubility Screening using Deep Neural Networks](https://mlsanity.com/paper/2203.10970/) (2022)
- [Classifying a specific image region using convolutional nets with an ROI mask as input](https://mlsanity.com/paper/1812.00291/) (2018)
- [BRDF Estimation of Complex Materials with Nested Learning](https://mlsanity.com/paper/1811.09131/) (2018)

## Built on by (most cited first, 5 of 5)

- [Shape and Texture Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2503.23062/) (2025)
- [Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data](https://mlsanity.com/paper/2403.03309/) (2024)
- [LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images](https://mlsanity.com/paper/2403.13171/) (2024)
- [SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art](https://mlsanity.com/paper/2511.01817/) (2025)
- [FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification](https://mlsanity.com/paper/2603.17390/) (2026)

## Abstract (arXiv)

Visual recognition of materials and their states is essential for understanding most aspects of the world, from determining whether food is cooked, metal is rusted, or a chemical reaction has occurred. However, current image recognition methods are limited to specific classes and properties and can't handle the vast number of material states in the world. To address this, we present MatSim: the first dataset and benchmark for computer vision-based recognition of similarities and transitions between materials and textures, focusing on identifying any material under any conditions using one or a few examples. The dataset contains synthetic and natural images. The synthetic images were rendered using giant collections of textures, objects, and environments generated by computer graphics artists. We use mixtures and gradual transitions between materials to allow the system to learn cases with smooth transitions between states (like gradually cooked food). We also render images with materials inside transparent containers to support beverage and chemistry lab use cases. We use this dataset to train a siamese net that identifies the same material in different objects, mixtures, and environments. The descriptor generated by this net can be used to identify the states of materials and their subclasses using a single image. We also present the first few-shot material recognition benchmark with images from a wide range of fields, including the state of foods and drinks, types of grounds, and many other use cases. We show that a net trained on the MatSim synthetic dataset outperforms state-of-the-art models like Clip on the benchmark and also achieves good results on other unsupervised material classification tasks.