# An Extensible Multimodal Multi-task Object Dataset with Materials

arXiv: https://arxiv.org/abs/2305.14352 · HTML: https://mlsanity.com/paper/2305.14352/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 29 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images](https://mlsanity.com/paper/1612.06341/) (2016)
- [Low-shot learning with large-scale diffusion](https://mlsanity.com/paper/1706.02332/) (2017)
- [A 4D Light-Field Dataset and CNN Architectures for Material Recognition](https://mlsanity.com/paper/1608.06985/) (2016)
- [Multi-View Matrix Completion for Multi-Label Image Classification](https://mlsanity.com/paper/1904.03901/) (2019)

## Built on by (most cited first, 1 of 1)

- [Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2510.01833/) (2025)

## Abstract (arXiv)

We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon's product-category taxonomy. We also design a comprehensive taxonomy of 182 physical materials (e.g., Plastic rightarrow Thermoplastic rightarrow Acrylic). Objects are annotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework to quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.