# KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

arXiv: https://arxiv.org/abs/2305.18373 · HTML: https://mlsanity.com/paper/2305.18373/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 28 May 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment](https://mlsanity.com/paper/2203.07190/) (2022)
- [Explicit Knowledge-based Reasoning for Visual Question Answering](https://mlsanity.com/paper/1511.02570/) (2015)
- [SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models](https://mlsanity.com/paper/2210.03794/) (2022)
- [MetaCLUE: Towards Comprehensive Visual Metaphors Research](https://mlsanity.com/paper/2212.09898/) (2022)
- [Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters](https://mlsanity.com/paper/2211.01979/) (2022)
- [Ask Me What You Need: Product Retrieval using Knowledge from GPT-3](https://mlsanity.com/paper/2207.02516/) (2022)
- [Scalable Deep Learning Logo Detection](https://mlsanity.com/paper/1803.11417/) (2018)

## Built on by (most cited first, 7 of 7)

- [AdsQA: Towards Advertisement Video Understanding](https://mlsanity.com/paper/2509.08621/) (2025)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models](https://mlsanity.com/paper/2405.20846/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook](https://mlsanity.com/paper/2509.14142/) (2025)
- [A3: Towards Advertising Aesthetic Assessment](https://mlsanity.com/paper/2603.24037/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)

## Abstract (arXiv)

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.