# From Coarse to Fine-grained Concept based Discrimination for Phrase Detection

arXiv: https://arxiv.org/abs/2112.03237 · HTML: https://mlsanity.com/paper/2112.03237/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment](https://mlsanity.com/paper/1903.11649/) (2019)
- [Conditional Image-Text Embedding Networks](https://mlsanity.com/paper/1711.08389/) (2017)
- [Attention Correctness in Neural Image Captioning](https://mlsanity.com/paper/1605.09553/) (2016)
- [Query-guided Regression Network with Context Policy for Phrase Grounding](https://mlsanity.com/paper/1708.01676/) (2017)
- [Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues](https://mlsanity.com/paper/1611.06641/) (2016)
- [Phrase Localization Without Paired Training Examples](https://mlsanity.com/paper/1908.07553/) (2019)
- [Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries](https://mlsanity.com/paper/1704.03944/) (2017)
- [Modularized Textual Grounding for Counterfactual Resilience](https://mlsanity.com/paper/1904.03589/) (2019)
- [ViCo: Word Embeddings from Visual Co-occurrences](https://mlsanity.com/paper/1908.08527/) (2019)
- [Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation](https://mlsanity.com/paper/1711.09509/) (2017)

## Abstract (arXiv)

Phrase detection requires methods to identify if a phrase is relevant to an image and localize it, if applicable. A key challenge for training more discriminative detection models is sampling negatives. Sampling techniques from prior work focus primarily on hard, often noisy, negatives disregarding the broader distribution of negative samples. Our proposed CFCD-Net addresses this through two novels methods. First, we generate groups of semantically similar words we call concepts (eg, dog, cat, horse and  car, truck, SUV), and then train our CFCD-Net to discriminate between a region of interest and its unrelated concepts. Second, for phrases containing fine-grained mutually-exclusive words (eg, colors), we force the model to select only one applicable phrase for each region using our novel fine-grained module (FGM). We evaluate our approach on Flickr30K Entities and RefCOCO+, where we improve mAP over the state-of-the-art by 1.5-2 points. When considering only the phrases affected by our FGM module, we improve by 3-4 points on both datasets.