# Context-Aware Group Captioning via Self-Attention and Contrastive Features

arXiv: https://arxiv.org/abs/2004.03708 · HTML: https://mlsanity.com/paper/2004.03708/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning](https://mlsanity.com/paper/2104.11056/) (2021)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images](https://mlsanity.com/paper/2110.00519/) (2021)
- [Group-based Distinctive Image Captioning with Memory Attention](https://mlsanity.com/paper/2108.09151/) (2021)
- [Semantic Alignment for Multimodal Large Language Models](https://mlsanity.com/paper/2408.12867/) (2024)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions](https://mlsanity.com/paper/2106.08543/) (2021)
- [COMICS: End-to-end Bi-grained Contrastive Learning for Multi-face Forgery Detection](https://mlsanity.com/paper/2308.01520/) (2023)
- [Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA](https://mlsanity.com/paper/2312.13594/) (2023)
- [Improving Reference-based Distinctive Image Captioning with Contrastive Rewards](https://mlsanity.com/paper/2306.14259/) (2023)
- [Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning](https://mlsanity.com/paper/2302.02124/) (2023)
- [Visual Commonsense based Heterogeneous Graph Contrastive Learning](https://mlsanity.com/paper/2311.06553/) (2023)
- [Maybe you are looking for CroQS: Cross-modal Query Suggestion for Text-to-Image Retrieval](https://mlsanity.com/paper/2412.13834/) (2024)
- [Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene](https://mlsanity.com/paper/2203.08362/) (2022)
- [ImageSet2Text: Describing Sets of Images through Text](https://mlsanity.com/paper/2503.19361/) (2025)
- [Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention](https://mlsanity.com/paper/2504.02496/) (2025)

## Abstract (arXiv)

While image captioning has progressed rapidly, existing works focus mainly on describing single images. In this paper, we introduce a new task, context-aware group captioning, which aims to describe a group of target images in the context of another group of related reference images. Context-aware group captioning requires not only summarizing information from both the target and reference image group but also contrasting between them. To solve this problem, we propose a framework combining self-attention mechanism with contrastive feature construction to effectively summarize common information from each image group while capturing discriminative information between them. To build the dataset for this task, we propose to group the images and generate the group captions based on single image captions using scene graphs matching. Our datasets are constructed on top of the public Conceptual Captions dataset and our new Stock Captions dataset. Experiments on the two datasets show the effectiveness of our method on this new task. Related Datasets and code are released at https://lizw14.github.io/project/groupcap .