# Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching

arXiv: https://arxiv.org/abs/1909.11416 · HTML: https://mlsanity.com/paper/1909.11416/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 30 of 49)

- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search](https://mlsanity.com/paper/2101.03036/) (2021)
- [Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering](https://mlsanity.com/paper/2104.14085/) (2021)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval](https://mlsanity.com/paper/2108.02417/) (2021)
- [Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval](https://mlsanity.com/paper/2210.08908/) (2022)
- [Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching](https://mlsanity.com/paper/2110.02623/) (2021)
- [DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation](https://mlsanity.com/paper/2209.01339/) (2022)
- [Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control](https://mlsanity.com/paper/2402.17535/) (2024)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [Rethinking Benchmarks for Cross-modal Image-text Retrieval](https://mlsanity.com/paper/2304.10824/) (2023)
- [Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination](https://mlsanity.com/paper/2308.04380/) (2023)
- [Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query](https://mlsanity.com/paper/2103.01654/) (2021)
- [Negative Pre-aware for Noisy Cross-modal Matching](https://mlsanity.com/paper/2312.05777/) (2023)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [Distance Matters in Human-Object Interaction Detection](https://mlsanity.com/paper/2207.01869/) (2022)
- [Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval](https://mlsanity.com/paper/2308.04343/) (2023)
- [Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment](https://mlsanity.com/paper/2503.06974/) (2025)
- [Towards Deconfounded Image-Text Matching with Causal Inference](https://mlsanity.com/paper/2408.12292/) (2024)
- [Referring Expression Instance Retrieval and A Strong End-to-End Baseline](https://mlsanity.com/paper/2506.18246/) (2025)
- [More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching](https://mlsanity.com/paper/2105.09597/) (2021)
- [Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features](https://mlsanity.com/paper/2309.14999/) (2023)
- [Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval](https://mlsanity.com/paper/2406.05620/) (2024)
- [Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval](https://mlsanity.com/paper/2109.05523/) (2021)
- [Understanding Attention for Vision-and-Language Tasks](https://mlsanity.com/paper/2208.08104/) (2022)
- [Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching](https://mlsanity.com/paper/2404.18114/) (2024)
- [GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning](https://mlsanity.com/paper/2410.15266/) (2024)
- [SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs](https://mlsanity.com/paper/2504.13172/) (2025)
- [Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study](https://mlsanity.com/paper/2301.05174/) (2023)

## Abstract (arXiv)

Learning semantic correspondence between image and text is significant as it bridges the semantic gap between vision and language. The key challenge is to accurately find and correlate shared semantics in image and text. Most existing methods achieve this goal by representing the shared semantic as a weighted combination of all the fragments (image regions or text words), where fragments relevant to the shared semantic obtain more attention, otherwise less. However, despite relevant ones contribute more to the shared semantic, irrelevant ones will more or less disturb it, and thus will lead to semantic misalignment in the correlation phase. To address this issue, we present a novel Bidirectional Focal Attention Network (BFAN), which not only allows to attend to relevant fragments but also diverts all the attention into these relevant fragments to concentrate on them. The main difference with existing works is they mostly focus on learning attention weight while our BFAN focus on eliminating irrelevant fragments from the shared semantic. The focal attention is achieved by pre-assigning attention based on inter-modality relation, identifying relevant fragments based on intra-modality relation and reassigning attention. Furthermore, the focal attention is jointly applied in both image-to-text and text-to-image directions, which enables to avoid preference to long text or complex image. Experiments show our simple but effective framework significantly outperforms state-of-the-art, with relative Recall@1 gains of 2.2% on both Flicr30K and MSCOCO benchmarks.