# FoodSAM: Any Food Segmentation

arXiv: https://arxiv.org/abs/2308.05938 · HTML: https://mlsanity.com/paper/2308.05938/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 79 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)

## Built on by (most cited first, 19 of 19)

- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [PTQ4SAM: Post-Training Quantization for Segment Anything](https://mlsanity.com/paper/2405.03144/) (2024)
- [CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Model](https://mlsanity.com/paper/2402.03631/) (2024)
- [PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentation](https://mlsanity.com/paper/2401.13051/) (2024)
- [OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation](https://mlsanity.com/paper/2404.01409/) (2024)
- [NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation](https://mlsanity.com/paper/2408.11787/) (2024)
- [UniUltra: Interactive Parameter-Efficient SAM2 for Universal Ultrasound Segmentation](https://mlsanity.com/paper/2511.15771/) (2025)
- [FoodMem: Near Real-time and Precise Food Video Segmentation](https://mlsanity.com/paper/2407.12121/) (2024)
- [Recognizing Multiple Ingredients in Food Images Using a Single-Ingredient Classification Model](https://mlsanity.com/paper/2401.14579/) (2024)
- [Exploring PCA-based feature representations of image pixels via CNN to enhance food image segmentation](https://mlsanity.com/paper/2411.01469/) (2024)
- [RecipeGen: A Benchmark for Real-World Recipe Image Generation](https://mlsanity.com/paper/2503.05228/) (2025)
- [Improving Food Image Recognition with Noisy Vision Transformer](https://mlsanity.com/paper/2503.18997/) (2025)
- [Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentation](https://mlsanity.com/paper/2507.17347/) (2025)
- [Knowledge distillation to effectively attain both region-of-interest and global semantics from an image where multiple objects appear](https://mlsanity.com/paper/2407.08257/) (2024)
- [A SAM based Tool for Semi-Automatic Food Annotation](https://mlsanity.com/paper/2410.19756/) (2024)
- [SARTM: Segment Any RGB Thermal Model with Language aided Distillation](https://mlsanity.com/paper/2505.01950/) (2025)
- [Weakly Supervised Food Image Segmentation using Vision Transformers and Segment Anything Model](https://mlsanity.com/paper/2509.19028/) (2025)
- [Food Image Segmentation with LLM-Derived Ingredient Labels and Multimodal Fusion](https://mlsanity.com/paper/2607.25820/) (2026)
- [ZMIS-SAM: Segment Anything Model Enhanced with Wavelet Transform for Zooplankton Microscopy Image Instance Segmentation](https://mlsanity.com/paper/2607.27585/) (2026)

## Abstract (arXiv)

In this paper, we explore the zero-shot capability of the Segment Anything Model (SAM) for food image segmentation. To address the lack of class-specific information in SAM-generated masks, we propose a novel framework, called FoodSAM. This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic segmentation quality. Besides, we recognize that the ingredients in food can be supposed as independent individuals, which motivated us to perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Drawing inspiration from the recent success of promptable segmentation, we also extend FoodSAM to promptable segmentation, supporting various prompt variants. Consequently, FoodSAM emerges as an all-encompassing solution capable of segmenting food items at multiple levels of granularity. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. Extensive experiments demonstrate the feasibility and impressing performance of FoodSAM, validating SAM's potential as a prominent and influential tool within the domain of food image segmentation. We release our code at https://github.com/jamesjg/FoodSAM.