# FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images

arXiv: https://arxiv.org/abs/2604.14388 · HTML: https://mlsanity.com/paper/2604.14388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 43 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale](https://mlsanity.com/paper/2412.05237/) (2024)
- [Learning Visual Grounding from Generative Vision and Language Model](https://mlsanity.com/paper/2407.14563/) (2024)
- [On Domain-Adaptive Post-Training for Multimodal Large Language Models](https://mlsanity.com/paper/2411.19930/) (2024)
- [Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models](https://mlsanity.com/paper/2308.03151/) (2023)
- [Regression in EO: Are VLMs Up to the Challenge?](https://mlsanity.com/paper/2502.14088/) (2025)
- [SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights](https://mlsanity.com/paper/2507.04412/) (2025)
- [January Food Benchmark (JFB): A Public Benchmark Dataset and Evaluation Suite for Multimodal Food Analysis](https://mlsanity.com/paper/2508.09966/) (2025)

## Abstract (arXiv)

Humans routinely infer taste, smell, texture, and even sound from food images a phenomenon well studied in cognitive science. However, prior vision language research on food has focused primarily on recognition tasks such as meal identification, ingredient detection, and nutrition estimation. Image-based prediction of multisensory experience remains largely unexplored. We introduce FoodSense, a human-annotated dataset for cross-sensory inference containing 66,842 participant-image pairs across 2,987 unique food images. Each pair includes numeric ratings (1-5) and free-text descriptors for four sensory dimensions: taste, smell, texture, and sound. To enable models to both predict and explain sensory expectations, we expand short human annotations into image-grounded reasoning traces. A large language model generates visual justifications conditioned on the image, ratings, and descriptors. Using these annotations, we train FoodSense-VL, a vision language benchmark model to produce both multisensory ratings and grounded explanations directly from food images. This work connects cognitive science findings on cross-sensory perception with modern instruction tuning for multimodal models and shows that many popular evaluation metrics are insufficient for visually sensory inference.