mlsanityData to 7 Oct 2026

Understanding Social Media Cross-Modality Discourse in Linguistic Space

arXiv 2302.13311 ↗First posted 26 Feb 2023cs.MM cs.CL cs.SIcited by 6

Not on the map. Why?

Abstract (arXiv)

The multimedia communications with texts and images are popular on social media. However, limited studies concern how images are structured with texts to form coherent meanings in human cognition. To fill in the gap, we present a novel concept of cross-modality discourse, reflecting how human readers couple image and text understandings. Text descriptions are first derived from images (named as subtitles) in the multimedia contexts. Five labels -- entity-level insertion, projection and concretization and scene-level restatement and extension -- are further employed to shape the structure of subtitles and texts and present their joint meanings. As a pilot study, we also build the very first dataset containing 16K multimedia tweets with manually annotated discourse labels. The experimental results show that the multimedia encoder based on multi-head attention with captions is able to obtain the-state-of-the-art results.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 26 Feb 2023.

Builds on

36 of 62 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2016Self-critical Sequence Training for Image Captioningcited by 544
  11. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  12. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
24 more
  1. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  2. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  3. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
  4. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  5. 2020COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twittercited by 96
  6. 2019A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Mediacited by 95
  7. 2015Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answeringcited by 66
  8. 2020Cross-media Structured Common Space for Multimedia Event Extractioncited by 59
  9. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 54
  10. 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
  11. 2021RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERcited by 37
  12. 2020Multimodal Categorization of Crisis Events in Social Mediacited by 34
  13. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  14. 2017Neural Discourse Structure for Text Categorizationcited by 24
  15. 2015Better Document-level Sentiment Analysis from RST Discourse Parsingcited by 20
  16. 2021Check It Again: Progressive Visual Question Answering via Visual Entailmentcited by 17
  17. 2019Adaptively Aligned Image Captioning via Adaptive Attention Timecited by 15
  18. 2019CITE: A Corpus of Image-Text Discourse Relationscited by 12
  19. 2020Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordingscited by 11
  20. 2019Microblog Hashtag Generation via Encoding Conversation Contextscited by 10
  21. 2021Point-of-Interest Type Prediction using Text and Imagescited by 10
  22. 2019Hierarchical Pointer Net Parsingcited by 8
  23. 2018Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignmentcited by 3
  24. 2018A Joint Model of Conversational Discourse and Latent Topics on Microblogscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  2. 2025RONA: Pragmatically Diverse Image Captioning with Coherence Relationscited by 3
  3. 2025IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detectioncited by 3
  4. 2023Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Taskscited by 2
  5. 2024ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesiscited by 0
  6. 2025From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generationcited by 0