mlsanityData to 7 Oct 2026

Semi-supervised multimodal coreference resolution in image narrations

arXiv 2310.13619 ↗First posted 20 Oct 2023cs.CL cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we study multimodal coreference resolution, specifically where a longer descriptive text, i.e., a narration is paired with an image. This poses significant challenges due to fine-grained image-text alignment, inherent ambiguity present in narrative language, and unavailability of large annotated training sets. To tackle these challenges, we present a data efficient semi-supervised approach that utilizes image-narration pairs to resolve coreferences and narrative grounding in a multimodal context. Our approach incorporates losses for both labeled and unlabeled data within a cross-modal framework. Our evaluation shows that the proposed approach outperforms strong baselines both quantitatively and qualitatively, for the tasks of coreference resolution and narrative grounding.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2015Fast R-CNNcited by 2,930
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
25 more
  1. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  10. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  11. 2016Visual Dialogcited by 377
  12. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  13. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  14. 2017End-to-end Neural Coreference Resolutioncited by 240
  15. 2019Connecting Vision and Language with Localized Narrativescited by 225
  16. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  17. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  18. 2018Neural Abstractive Text Summarization with Sequence-to-Sequence Modelscited by 39
  19. 2021Adaptive Consistency Regularization for Semi-Supervised Transfer Learningcited by 37
  20. 2023Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequencescited by 31
  21. 2021On Generalization in Coreference Resolutioncited by 24
  22. 2021Moving on from OntoNotes: Coreference Resolution Model Transfercited by 18
  23. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  24. 2022Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolutioncited by 6
  25. 2023Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspectivecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025MDC-R: The Minecraft Dialogue Corpus with Referencecited by 2
  2. 2025Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structurescited by 0
  3. 2026One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognitioncited by 0
  4. 2026Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Modelcited by 0