# Multimodal Coreference Resolution for Chinese Social Media Dialogues: Dataset and Benchmark Approach

arXiv: https://arxiv.org/abs/2504.14321 · HTML: https://mlsanity.com/paper/2504.14321/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 19 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [CCMB: A Large-scale Chinese Cross-modal Benchmark](https://mlsanity.com/paper/2205.03860/) (2022)
- [Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://mlsanity.com/paper/2407.21489/) (2024)
- [What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues](https://mlsanity.com/paper/1909.00421/) (2019)
- [OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres](https://mlsanity.com/paper/2106.00933/) (2021)
- [Generating Descriptions with Grounded and Co-Referenced People](https://mlsanity.com/paper/1704.01518/) (2017)
- [Seq2seq is All You Need for Coreference Resolution](https://mlsanity.com/paper/2310.13774/) (2023)
- [Who are you referring to? Coreference resolution in image narrations](https://mlsanity.com/paper/2211.14563/) (2022)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolution](https://mlsanity.com/paper/2403.19259/) (2024)
- [Resolving References in Visually-Grounded Dialogue via Text Generation](https://mlsanity.com/paper/2309.13430/) (2023)

## Built on by (most cited first, 1 of 1)

- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)

## Abstract (arXiv)

Multimodal coreference resolution (MCR) aims to identify mentions referring to the same entity across different modalities, such as text and visuals, and is essential for understanding multimodal content. In the era of rapidly growing mutimodal content and social media, MCR is particularly crucial for interpreting user interactions and bridging text-visual references to improve communication and personalization. However, MCR research for real-world dialogues remains unexplored due to the lack of sufficient data resources. To address this gap, we introduce TikTalkCoref, the first Chinese multimodal coreference dataset for social media in real-world scenarios, derived from the popular Douyin short-video platform. This dataset pairs short videos with corresponding textual dialogues from user comments and includes manually annotated coreference clusters for both person mentions in the text and the coreferential person head regions in the corresponding video frames. We also present an effective benchmark approach for MCR, focusing on the celebrity domain, and conduct extensive experiments on our dataset, providing reliable benchmark results for this newly constructed dataset. We will release the TikTalkCoref dataset to facilitate future research on MCR for real-world social media dialogues.