mlsanityData to 7 Oct 2026

Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval

arXiv 2204.09730 ↗First posted 20 Apr 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Cross-modal image-recipe retrieval has gained significant attention in recent years. Most work focuses on improving cross-modal embeddings using unimodal encoders, that allow for efficient retrieval in large-scale databases, leaving aside cross-attention between modalities which is more computationally expensive. We propose a new retrieval framework, T-Food (Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval) that exploits the interaction between modalities in a novel regularization scheme, while using only unimodal encoders at test time for efficient retrieval. We also capture the intra-dependencies between recipe entities with a dedicated recipe encoder, and propose new variants of triplet losses with dynamic margins that adapt to the difficulty of the task. Finally, we leverage the power of the recent Vision and Language Pretraining (VLP) models such as CLIP for the image encoder. Our approach outperforms existing approaches by a large margin on the Recipe1M dataset. Specifically, we achieve absolute improvements of 8.1 % (72.6 R@1) and +10.9 % (44.6 R@1) on the 1k and 10k test sets respectively. The code is available here:https://github.com/mshukor/TFood

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

51 of 70 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
38 more
  1. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  12. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 784
  13. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  14. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  15. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2015Skip-Thought Vectorscited by 321
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  20. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  21. 2019Cross-Batch Memory for Embedding Learningcited by 130
  22. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  23. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  24. 2021Solving Inefficiency of Self-supervised Representation Learningcited by 41
  25. 2019Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Imagescited by 38
  26. 2018Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddingscited by 35
  27. 2021Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learningcited by 28
  28. 2015Deep Feature Learning with Relative Distance Comparison for Person Re-identificationcited by 25
  29. 2020Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanismcited by 20
  30. 2021Learning Structural Representations for Recipe Generation and Food Retrievalcited by 17
  31. 2020MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Modelcited by 16
  32. 2018Learning Incremental Triplet Margin for Person Re-identificationcited by 9
  33. 2019A weakly supervised adaptive triplet loss for deep metric learningcited by 9
  34. 2020Cross-Modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Representation Learningcited by 9
  35. 2017Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Healthcited by 8
  36. 2021Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineeringcited by 8
  37. 2019Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTAcited by 7
  38. 2021Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Servicecited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

6 papers.

  1. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  2. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrievalcited by 3
  3. 2023MALM: Mask Augmentation based Local Matching for Food-Recipe Retrievalcited by 1
  4. 2025Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrievalcited by 1
  5. 2025Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrievalcited by 0
  6. 2026SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embeddingcited by 0