mlsanityData to 14 Oct 2026

Aligning VLM Assistants with Personalized Situated Cognition

arXiv 2506.00930 ↗First posted 1 Jun 2025cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) aligned with general human objectives, such as being harmless and hallucination-free, have become valuable assistants of humans in managing visual tasks. However, people with diversified backgrounds have different cognition even in the same situation. Consequently, they may have personalized expectations for VLM assistants. This highlights the urgent need to align VLM assistants with personalized situated cognition for real-world assistance. To study this problem, we first simplify it by characterizing individuals based on the sociological concept of Role-Set. Then, we propose to evaluate the individuals' actions to examine whether the personalized alignment is achieved. Further, we construct a benchmark named PCogAlignBench, which includes 18k instances and 20 individuals with different Role-Sets. Finally, we present a framework called PCogAlign, which constructs a cognition-aware and action-based reward model for personalized alignment. Experimental results and human evaluations demonstrate the reliability of the PCogAlignBench and the effectiveness of our proposed PCogAlign. We will open-source the constructed benchmark and code at https://github.com/NLPGM/PCogAlign.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 61 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

40 of 61 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Personalized Multimodal Large Language Models: A Survey3 Dec
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
  4. 2024Self-Supervised Visual Preference Alignment16 Apr
  5. 2024Assessment of Multimodal Large Language Models in Alignment with Human Values26 Mar
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 33

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  10. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  11. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
21 more
  1. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  2. 2024A Roadmap to Pluralistic Alignmentcited by 184
  3. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  4. 2024Personalization of Large Language Models: A Surveycited by 104
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  6. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  7. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  8. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  9. 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 49
  10. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 43
  11. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  12. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
  13. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
  14. 2024AI PERSONA: Towards Life-long Personalization of LLMscited by 32
  15. 2024Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyondcited by 25
  16. 2024Self-Refine Instruction-Tuning for Aligning Reasoning in Language Modelscited by 24
  17. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  18. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  19. 2024MAP: Multi-Human-Value Alignment Palettecited by 20
  20. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  21. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Timecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captionscited by 5
  2. 2026Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025LVLM-Aided Alignment of Task-Specific Vision Models
  2. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  3. 2026EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
  4. 2026Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment
  5. 2025Improving Alignment in LVLMs with Debiased Self-Judgment
  6. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  7. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM
  8. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models