# SoMeLVLM: A Large Vision Language Model for Social Media Processing

arXiv: https://arxiv.org/abs/2402.13022 · HTML: https://mlsanity.com/paper/2402.13022/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 75 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Valley: Video Assistant with Large Language model Enhanced abilitY](https://mlsanity.com/paper/2306.07207/) (2023)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://mlsanity.com/paper/2010.03640/) (2020)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [A Large Self-Annotated Corpus for Sarcasm](https://mlsanity.com/paper/1704.05579/) (2017)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Stance and Sentiment in Tweets](https://mlsanity.com/paper/1605.01655/) (2016)
- [Modeling Empathy and Distress in Reaction to News Stories](https://mlsanity.com/paper/1808.10399/) (2018)
- [Humor Detection: A Transformer Gets the Last Laugh](https://mlsanity.com/paper/1909.00252/) (2019)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)
- [Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines](https://mlsanity.com/paper/2104.08790/) (2021)
- [Automatically Identifying Complaints in Social Media](https://mlsanity.com/paper/1906.03890/) (2019)
- [Quantifying Intimacy in Language](https://mlsanity.com/paper/2011.03020/) (2020)
- [Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica](https://mlsanity.com/paper/2109.02738/) (2021)
- [SemEval-2020 Task 7: Assessing Humor in Edited News Headlines](https://mlsanity.com/paper/2008.00304/) (2020)

## Built on by (most cited first, 11 of 11)

- [SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users](https://mlsanity.com/paper/2504.10157/) (2025)
- [ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents](https://mlsanity.com/paper/2410.20746/) (2024)
- [INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance](https://mlsanity.com/paper/2406.09105/) (2024)
- [Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding](https://mlsanity.com/paper/2502.09906/) (2025)
- [SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Services](https://mlsanity.com/paper/2505.23065/) (2025)
- [MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning](https://mlsanity.com/paper/2510.17590/) (2025)
- [Overview of the CAIL 2023 Argument Mining Track](https://mlsanity.com/paper/2406.14503/) (2024)
- [Fine-grained Verbal Attack Detection via a Hierarchical Divide-and-Conquer Framework](https://mlsanity.com/paper/2601.06907/) (2026)
- [SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context](https://mlsanity.com/paper/2606.26654/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities.