# New York Smells: A Large Multimodal Dataset for Olfaction

arXiv: https://arxiv.org/abs/2511.20544 · HTML: https://mlsanity.com/paper/2511.20544/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Molecules](https://mlsanity.com/paper/1910.10685/) (2019)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Connecting Look and Feel: Associating the visual and tactile properties of physical materials](https://mlsanity.com/paper/1704.03822/) (2017)
- [SmellNet: A Large-scale Dataset for Real-world Smell Recognition](https://mlsanity.com/paper/2506.00239/) (2025)
- [Learning to Taste: A Multimodal Wine Dataset](https://mlsanity.com/paper/2308.16900/) (2023)
- [Hierarchical Material Recognition from Local Appearance](https://mlsanity.com/paper/2505.22911/) (2025)

## Built on by (most cited first, 3 of 3)

- [See & Sniff: Learning Visuo-Olfactory Representations](https://mlsanity.com/paper/2606.27307/) (2026)
- [What Images Cannot Say: Language-Guided Olfactory Representation Learning](https://mlsanity.com/paper/2607.06402/) (2026)
- [Sense it with your eyes: Sensation Generation and Understanding for Advertisements](https://mlsanity.com/paper/2607.25314/) (2026)

## Abstract (arXiv)

While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key bottleneck is the lack of diverse, multimodal olfactory training data collected in natural settings. We present New York Smells, a large dataset of paired image and olfactory signals captured ``in the wild.'' Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately 70× more objects than existing olfactory datasets. Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. Through experiments on our dataset, we find that visual data enables cross-modal olfactory representation learning, and that our learned olfactory representations outperform widely-used hand-crafted features.