# F4-ITS: Fine-grained Feature Fusion for Food Image-Text Search

arXiv: https://arxiv.org/abs/2508.17037 · HTML: https://mlsanity.com/paper/2508.17037/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Zero-shot Composed Text-Image Retrieval](https://mlsanity.com/paper/2306.07272/) (2023)
- [Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity](https://mlsanity.com/paper/2409.04918/) (2024)
- [Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation](https://mlsanity.com/paper/2505.21549/) (2025)
- [PDV: Prompt Directional Vectors for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2502.07215/) (2025)
- [Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment](https://mlsanity.com/paper/2503.09081/) (2025)
- [ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning](https://mlsanity.com/paper/2410.17779/) (2024)

## Abstract (arXiv)

The proliferation of digital food content has intensified the need for robust and accurate systems capable of fine-grained visual understanding and retrieval. In this work, we address the challenging task of food image-to-text matching, a critical component in applications such as dietary monitoring, smart kitchens, and restaurant automation. We propose F4-ITS: Fine-grained Feature Fusion for Food Image-Text Search, a training-free, vision-language model (VLM)-guided framework that significantly improves retrieval performance through enhanced multi-modal feature representations. Our approach introduces two key contributions: (1) a uni-directional(and bi-directional) multi-modal fusion strategy that combines image embeddings with VLM-generated textual descriptions to improve query expressiveness, and (2) a novel feature-based re-ranking mechanism for top-k retrieval, leveraging predicted food ingredients to refine results and boost precision. Leveraging open-source image-text encoders, we demonstrate substantial gains over standard baselines - achieving ~10% and ~7.7% improvements in top-1 retrieval under dense and sparse caption scenarios, and a ~28.6% gain in top-k ingredient-level retrieval. Additionally, we show that smaller models (e.g., ViT-B/32) can match or outperform larger counterparts (e.g., ViT-H, ViT-G, ViT-bigG) when augmented with textual fusion, highlighting the effectiveness of our method in resource-constrained settings. Code and test datasets will be made publicly available at: https://github.com/mailcorahul/f4-its