mlsanityData to 7 Oct 2026

MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms

arXiv 2402.14154 ↗First posted 21 Feb 2024cs.CL cs.CV cs.CYcited by 26

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. Multimodal Large Language Models (MLLMs) have emerged as a promising solution to these challenges, yet they struggle to accurately interpret human emotions and complex content such as misinformation. This paper introduces MM-Soc, a comprehensive benchmark designed to evaluate MLLMs' understanding of multimodal social media content. MM-Soc compiles prominent multimodal datasets and incorporates a novel large-scale YouTube tagging dataset, targeting a range of tasks from misinformation detection, hate speech detection, and social context generation. Through our exhaustive evaluation on ten size-variants of four open-source MLLMs, we have identified significant performance disparities, highlighting the need for advancements in models' social understanding capabilities. Our analysis reveals that, in a zero-shot setting, various types of MLLMs generally exhibit difficulties in handling social media tasks. However, MLLMs demonstrate performance improvements post fine-tuning, suggesting potential pathways for improvement. Our code and data are available at https://github.com/claws-lab/MMSoc.git.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 80 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

38 of 80 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
22 more
  1. 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
  2. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  3. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  4. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  5. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  6. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  7. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 77
  8. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  9. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  10. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 61
  11. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  12. 2018On the Origins of Memes by Means of Fringe Web Communitiescited by 45
  13. 2020Racism is a Virus: Anti-Asian Hate and Counterspeech in Social Media during the COVID-19 Crisiscited by 40
  14. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 39
  15. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  16. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Modelscited by 23
  17. 2023AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graphcited by 17
  18. 2023Predicting Information Pathways Across Online Communitiescited by 15
  19. 2023PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learnerscited by 13
  20. 2023The Good, The Bad, and Why: Unveiling Emotions in Generative AIcited by 10
  21. 2023Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Modelscited by 6
  22. 2023A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creatorscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 12; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  7. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  10. 2024MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Modelscited by 20
  11. 2024UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Modelscited by 18
  12. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
14 more
  1. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  2. 2024Native vs Non-Native Language Prompting: A Comparative Analysiscited by 9
  3. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  4. 2025LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loopscited by 9
  5. 2025am-ELO: A Stable Framework for Arena-based LLM Evaluationcited by 8
  6. 2025Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluationcited by 7
  7. 2025XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMscited by 6
  8. 2025Commander-GPT: Fully Unleashing the Sarcasm Detection Capability of Multi-Modal Large Language Modelscited by 3
  9. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  10. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  11. 2026Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGCcited by 2
  12. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Predictioncited by 1
  13. 2026RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderationcited by 1
  14. 2024Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
  2. 2025Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models
  3. 2026Is AI Ready for Multimodal Hate Speech Detection? A Comprehensive Dataset and Benchmark Evaluation ↗
  4. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
  5. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  6. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge
  7. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  8. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse