# Self-Enhancement Improves Text-Image Retrieval in Foundation Visual-Language Models

arXiv: https://arxiv.org/abs/2306.06691 · HTML: https://mlsanity.com/paper/2306.06691/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 9 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 4 of 4)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Localization Guided Learning for Pedestrian Attribute Recognition](https://mlsanity.com/paper/1808.09102/) (2018)
- [UPAR: Unified Pedestrian Attribute Recognition and Person Retrieval](https://mlsanity.com/paper/2209.02522/) (2022)

## Built on by (most cited first, 1 of 1)

- [The Solution for the CVPR2024 NICE Image Captioning Challenge](https://mlsanity.com/paper/2404.12739/) (2024)

## Abstract (arXiv)

The emergence of cross-modal foundation models has introduced numerous approaches grounded in text-image retrieval. However, on some domain-specific retrieval tasks, these models fail to focus on the key attributes required. To address this issue, we propose a self-enhancement framework, A3R, based on the CLIP-ViT/G-14, one of the largest cross-modal models. First, we perform an Attribute Augmentation strategy to enrich the textual description for fine-grained representation before model learning. Then, we propose an Adaption Re-ranking method to unify the representation space of textual query and candidate images and re-rank candidate images relying on the adapted query after model learning. The proposed framework is validated to achieve a salient improvement over the baseline and other teams' solutions in the cross-modal image retrieval track of the 1st foundation model challenge without introducing any additional samples. The code is available at urlhttps://github.com/CapricornGuang/A3R.