# Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health

arXiv: https://arxiv.org/abs/1702.06318 · HTML: https://mlsanity.com/paper/1702.06318/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2017.

## Built on by (most cited first, 8 of 8)

- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2204.09730/) (2022)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [Othering and low status framing of immigrant cuisines in US restaurant reviews and large language models](https://mlsanity.com/paper/2307.07645/) (2023)
- [Formation of Social Ties Influences Food Choice: A Campus-Wide Longitudinal Study](https://mlsanity.com/paper/2102.08755/) (2021)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)
- [Comfort Foods and Community Connectedness: Investigating Diet Change during COVID-19 Using YouTube Videos on Twitter](https://mlsanity.com/paper/2305.11398/) (2023)

## Abstract (arXiv)

Food is an integral part of our life and what and how much we eat crucially affects our health. Our food choices largely depend on how we perceive certain characteristics of food, such as whether it is healthy, delicious or if it qualifies as a salad. But these perceptions differ from person to person and one person's "single lettuce leaf" might be another person's "side salad". Studying how food is perceived in relation to what it actually is typically involves a laboratory setup. Here we propose to use recent advances in image recognition to tackle this problem. Concretely, we use data for 1.9 million images from Instagram from the US to look at systematic differences in how a machine would objectively label an image compared to how a human subjectively does. We show that this difference, which we call the "perception gap", relates to a number of health outcomes observed at the county level. To the best of our knowledge, this is the first time that image recognition is being used to study the "misalignment" of how people describe food images vs. what they actually depict.