mlsanityData to 7 Oct 2026

FETA: Towards Specializing Foundation Models for Expert Task Applications

arXiv 2209.03648 ↗First posted 8 Sep 2022cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e.g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training. This underlines the necessity to explicitly evaluate and finetune FMs on such expert tasks, arguably ones that appear the most in practical real-world applications. In this paper, we propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions. Our FETA benchmark focuses on text-to-image and image-to-text retrieval in public car manuals and sales catalogue brochures. FETA is equipped with a procedure for completely automatic annotation extraction (code would be released upon acceptance), allowing easy extension of FETA to more documentation types and application domains in the future. Our automatic annotation leads to an automated performance metric shown to be consistent with metrics computed on human-curated annotations (also released). We provide multiple baselines and analysis of popular FMs on FETA leading to several interesting findings that we believe would be very valuable to the FM community, paving the way towards real-world application of FMs for practical expert tasks currently 'overlooked' by standard benchmarks focusing on common objects.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 48 of its 81 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

48 of 81 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
36 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2022A Generalist Agentcited by 802
  11. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  13. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  16. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  17. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  18. 2021InfographicVQAcited by 430
  19. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  20. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  21. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  22. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  23. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  24. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  25. 2015Order-Embeddings of Images and Languagecited by 112
  26. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  27. 2022TableFormer: Table Structure Understanding with Transformerscited by 93
  28. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  29. 2019Patent Claim Generation by Fine-Tuning OpenAI GPT-2cited by 49
  30. 2019Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detectioncited by 31
  31. 2018A flexible model for training action localization with varying levels of supervisioncited by 16
  32. 2020Learning Object Detection from Captions via Textual Scene Attributescited by 14
  33. 2019Accurate Visual Localization for Automotive Applicationscited by 10
  34. 2021Robust PDF Document Conversion Using Recurrent Neural Networkscited by 9
  35. 2022Delivering Document Conversion as a Cloud Service with High Throughput and Responsivenesscited by 7
  36. 2017Learning from Video and Text via Large-Scale Discriminative Clusteringcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 3; 2025: 1.

13 papers.

Sort:

By area:

  1. 2023Class-Incremental Learning: A Surveycited by 209
  2. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  4. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  5. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  6. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  7. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  8. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  9. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  10. 2024Geospatial foundation models for image analysis: evaluating and enhancing NASA-IBM Prithvi's domain adaptabilitycited by 11
  11. 2023ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Datacited by 9
  12. 2023Enhancing Plasticity for First Session Adaptation Continual Learningcited by 2
1 more
  1. 2025A Visual RAG Pipeline for Few-Shot Fine-Grained Product Classificationcited by 1