mlsanityData to 14 Oct 2026

SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Services

arXiv 2505.23065 ↗First posted 29 May 2025cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the increasing integration of visual and textual content in Social Networking Services (SNS), evaluating the multimodal capabilities of Large Language Models (LLMs) is crucial for enhancing user experience, content understanding, and platform intelligence. Existing benchmarks primarily focus on text-centric tasks, lacking coverage of the multimodal contexts prevalent in modern SNS ecosystems. In this paper, we introduce SNS-Bench-VL, a comprehensive multimodal benchmark designed to assess the performance of Vision-Language LLMs in real-world social media scenarios. SNS-Bench-VL incorporates images and text across 8 multimodal tasks, including note comprehension, user engagement analysis, information retrieval, and personalized recommendation. It comprises 4,001 carefully curated multimodal question-answer pairs, covering single-choice, multiple-choice, and open-ended tasks. We evaluate over 25 state-of-the-art multimodal LLMs, analyzing their performance across tasks. Our findings highlight persistent challenges in multimodal social context comprehension. We hope SNS-Bench-VL will inspire future research towards robust, context-aware, and human-aligned multimodal intelligence for next-generation social networking services.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 94 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

79 of 94 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024A Survey on Evaluation of Multimodal Large Language Models28 Aug
  5. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models5 Aug
  6. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  7. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  8. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?14 Jun
  9. 2024Needle In A Multimodal Haystack11 Jun
  10. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models9 Jun
  11. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  12. 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction24 Apr
10 more
  1. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  2. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms21 Feb
  3. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  4. 2024SoMeLVLM: A Large Vision Language Model for Social Media Processing20 Feb
  5. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023Generating Images with Multimodal Language Models26 May

Other references · 57

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2022Galactica: A Large Language Model for Sciencecited by 744
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  11. 2025Kimi-VL Technical Reportcited by 388
  12. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  13. 2022Visual Spatial Reasoningcited by 293
  14. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  15. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  16. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  17. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  18. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  19. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  20. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  21. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  22. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  23. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  24. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  25. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  26. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
  27. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  28. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 65
  29. 2024Red Teaming Visual Language Modelscited by 65
  30. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  31. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 52
  32. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 46
  33. 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 42
  34. 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMscited by 33
  35. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  36. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  37. 2024MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistencycited by 22
  38. 2020Document Visual Question Answering Challenge 2020cited by 20

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Fine-grained Verbal Attack Detection via a Hierarchical Divide-and-Conquer Frameworkcited by 0
  2. 2026World Model-Guided Reinforcement Learning via Counterfactual User Engagement Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMs
  2. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  4. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
  5. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  6. 2024A Survey on Benchmarks of Multimodal Large Language Models
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  8. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models