# Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding

arXiv: https://arxiv.org/abs/2212.06971 · HTML: https://mlsanity.com/paper/2212.06971/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 40 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019)
- [A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension](https://mlsanity.com/paper/1909.07072/) (2019)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)
- [Transforming Question Answering Datasets Into Natural Language Inference Datasets](https://mlsanity.com/paper/1809.02922/) (2018)
- [Referring Expression Comprehension: A Survey of Methods and Datasets](https://mlsanity.com/paper/2007.09554/) (2020)
- [CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions](https://mlsanity.com/paper/1901.00850/) (2019)

## Built on by (most cited first, 4 of 4)

- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Advancing Visual Grounding with Scene Knowledge: Benchmark and Method](https://mlsanity.com/paper/2307.11558/) (2023)
- [MDC-R: The Minecraft Dialogue Corpus with Reference](https://mlsanity.com/paper/2506.22062/) (2025)
- [DIVE: Towards Descriptive and Diverse Visual Commonsense Generation](https://mlsanity.com/paper/2408.08021/) (2024)

## Abstract (arXiv)

From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions. We further create a benchmark, HumanCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes. We set up a context-object-aware method as a strong baseline that outperforms previous pre-trained and non-pretrained models. Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works. Data and code will be available https://github.com/Hxyou/HumanCog.