# Word2Pix: Word to Pixel Cross Attention Transformer in Visual Grounding

arXiv: https://arxiv.org/abs/2108.00205 · HTML: https://mlsanity.com/paper/2108.00205/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 59 references that are in the map.
- First posted on arXiv 31 Jul 2021.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)

## Built on by (most cited first, 16 of 16)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Language Adaptive Weight Generation for Multi-task Visual Grounding](https://mlsanity.com/paper/2306.04652/) (2023)
- [OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling](https://mlsanity.com/paper/2410.08021/) (2024)
- [ScanFormer: Referring Expression Comprehension by Iteratively Scanning](https://mlsanity.com/paper/2406.18048/) (2024)
- [Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2302.08641/) (2023)
- [ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax](https://mlsanity.com/paper/2303.01615/) (2023)
- [Towards Complex-query Referring Image Segmentation: A Novel Benchmark](https://mlsanity.com/paper/2309.17205/) (2023)
- [WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2505.18686/) (2025)
- [Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension](https://mlsanity.com/paper/2204.09957/) (2022)
- [SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Grounding](https://mlsanity.com/paper/2502.16786/) (2025)
- [Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models](https://mlsanity.com/paper/2404.12920/) (2024)
- [A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension](https://mlsanity.com/paper/2204.07913/) (2022)
- [Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding](https://mlsanity.com/paper/2410.23570/) (2024)
- [A Simple and Better Baseline for Visual Grounding](https://mlsanity.com/paper/2510.10587/) (2025)
- [LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension](https://mlsanity.com/paper/2511.12020/) (2025)
- [Transformer-based Spatial Grounding: A Comprehensive Survey](https://mlsanity.com/paper/2507.12739/) (2025)

## Abstract (arXiv)

Current one-stage methods for visual grounding encode the language query as one holistic sentence embedding before fusion with visual feature. Such a formulation does not treat each word of a query sentence on par when modeling language to visual attention, therefore prone to neglect words which are less important for sentence embedding but critical for visual grounding. In this paper we propose Word2Pix: a one-stage visual grounding network based on encoder-decoder transformer architecture that enables learning for textual to visual feature correspondence via word to pixel attention. The embedding of each word from the query sentence is treated alike by attending to visual pixels individually instead of single holistic sentence embedding. In this way, each word is given equivalent opportunity to adjust the language to vision attention towards the referent target through multiple stacks of transformer decoder layers. We conduct the experiments on RefCOCO, RefCOCO+ and RefCOCOg datasets and the proposed Word2Pix outperforms existing one-stage methods by a notable margin. The results obtained also show that Word2Pix surpasses two-stage visual grounding models, while at the same time keeping the merits of one-stage paradigm namely end-to-end training and real-time inference speed intact.