mlsanityData to 7 Oct 2026

Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models

arXiv 2606.11074 ↗First posted 9 Jun 2026cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the widespread deployment of Multimodal Large Language Models (MLLMs) in social interaction, understanding and controlling their behavior under complex personality conditions is essential. This paper introduces explicit personality conditioning and establishes a systematic evaluation framework encompassing single-personality induction, multi-personality induction, and personality switching. Experiments show that personality induction improves image captioning performance but can impair performance on tasks requiring precise reasoning, such as visual question answering (VQA). Balancing and residual effects are observed during multi-trait composition and dynamic switching, indicating that model behavior is co-modulated by both previous and current personality constraints. Existing prompt-based personality induction methods show limited transferability to multimodal settings. Our work reveals the dynamic and complex nature of personality modeling in MLLMs and underscores the need for robust, tailored methods for personality induction and evaluation. The code will be released when the paper is accepted.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 42 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

27 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  12. 2015Stacked Attention Networks for Image Question Answeringcited by 273
12 more
  1. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  2. 2015Exploring Models and Data for Image Question Answeringcited by 139
  3. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  4. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 49
  5. 2024Personality Alignment of Large Language Modelscited by 36
  6. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 23
  7. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  8. 2025Personality Vector: Modulating Personality of Large Language Models by Model Mergingcited by 16
  9. 2025Persona-Assigned Large Language Models Exhibit Human-Like Motivated Reasoningcited by 12
  10. 2026Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoningcited by 9
  11. 2025Scaling Personality Control in LLMs with Big Five Scaler Promptscited by 4
  12. 2026TabSieve: Explicit In-Table Evidence Selection for Tabular Predictioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselines
  2. 2024Extroversion or Introversion? Controlling The Personality of Your Large Language Models
  3. 2026PersonaVLM: Long-Term Personalized Multimodal LLMs
  4. 2026Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?
  5. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
  6. 2026PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems
  7. 2026Knowing You at First Glance: Inferring Apparent Personality from Faces
  8. 2024Assessing Social Alignment: Do Personality-Prompted Large Language Models Behave Like Humans?