# Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication

arXiv: https://arxiv.org/abs/2401.08212 · HTML: https://mlsanity.com/paper/2401.08212/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms](https://mlsanity.com/paper/2310.05984/) (2023)
- [SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning](https://mlsanity.com/paper/2307.10234/) (2023)
- [Sentiment of Emojis](https://mlsanity.com/paper/1509.07761/) (2015)
- [Spice up Your Chat: The Intentions and Sentiment Effects of Using Emoji](https://mlsanity.com/paper/1703.02860/) (2017)

## Built on by (most cited first, 8 of 8)

- [Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction](https://mlsanity.com/paper/2502.06882/) (2025)
- [Representation Bias in Political Sample Simulations with Large Language Models](https://mlsanity.com/paper/2407.11409/) (2024)
- [Emojis Decoded: Leveraging ChatGPT for Enhanced Understanding in Social Media Communications](https://mlsanity.com/paper/2402.01681/) (2024)
- [Irony in Emojis: A Comparative Study of Human and LLM Interpretation](https://mlsanity.com/paper/2501.11241/) (2025)
- [Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration](https://mlsanity.com/paper/2404.09690/) (2024)
- [When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicity](https://mlsanity.com/paper/2509.11141/) (2025)
- [Semantics Preserving Emoji Recommendation with Large Language Models](https://mlsanity.com/paper/2409.10760/) (2024)
- [The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation](https://mlsanity.com/paper/2506.00583/) (2025)

## Abstract (arXiv)

Leveraging Large Multimodal Models (LMMs) to simulate human behaviors when processing multimodal information, especially in the context of social media, has garnered immense interest due to its broad potential and far-reaching implications. Emojis, as one of the most unique aspects of digital communication, are pivotal in enriching and often clarifying the emotional and tonal dimensions. Yet, there is a notable gap in understanding how these advanced models, such as GPT-4V, interpret and employ emojis in the nuanced context of online interaction. This study intends to bridge this gap by examining the behavior of GPT-4V in replicating human-like use of emojis. The findings reveal a discernible discrepancy between human and GPT-4V behaviors, likely due to the subjective nature of human interpretation and the limitations of GPT-4V's English-centric training, suggesting cultural biases and inadequate representation of non-English cultures.