mlsanityData to 14 Oct 2026

Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas

arXiv 2410.03181 ↗First posted 4 Oct 2024cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This study is the first to explore whether multi-modal large language models (LLMs) can align their behaviors with visual personas, addressing a significant gap in the literature that predominantly focuses on text-based personas. We developed a novel dataset of 5K fictional avatar images for assignment as visual personas to LLMs, and analyzed their negotiation behaviors based on the visual traits depicted in these images, with a particular focus on aggressiveness. The results indicate that LLMs assess the aggressiveness of images in a manner similar to humans and output more aggressive negotiation behaviors when prompted with an aggressive visual persona. Interestingly, the LLM exhibited more aggressive negotiation behaviors when the opponent's image appeared less aggressive than their own, and less aggressive behaviors when the opponents image appeared more aggressive.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 33 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

11 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 10

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  4. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  5. 2023Aligning Language Models to User Opinionscited by 99
  6. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 92
  7. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 49
  8. 2022Let's Negotiate! A Survey of Negotiation Dialogue Systemscited by 31
  9. 2024LLMs with Personalities in Multi-issue Negotiation Gamescited by 22
  10. 2023Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactionscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Boosting Relational Deep Learning with Pretrained Tabular Modelscited by 9
  2. 2025A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMscited by 1
  3. 2026Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Sciencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models
  2. 2026What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors
  3. 2025Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans
  4. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
  5. 2024The influence of persona and conversational task on social interactions with a LLM-controlled embodied conversational agent
  6. 2026Self-reported archetypes and behavioral failures in Large Language Models
  7. 2023Intentional Biases in LLM Responses
  8. 2024Assessing Social Alignment: Do Personality-Prompted Large Language Models Behave Like Humans?