# Demographic User Modeling for Social Robotics with Multimodal Pre-trained Models

arXiv: https://arxiv.org/abs/2502.10642 · HTML: https://mlsanity.com/paper/2502.10642/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [General Facial Representation Learning in a Visual-Linguistic Manner](https://mlsanity.com/paper/2112.03109/) (2021)
- [Finetune like you pretrain: Improved finetuning of zero-shot vision models](https://mlsanity.com/paper/2212.00638/) (2022)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Large Multimodal Agents: A Survey](https://mlsanity.com/paper/2402.15116/) (2024)
- [User-LLM: Efficient LLM Contextualization with User Embeddings](https://mlsanity.com/paper/2402.13598/) (2024)
- [User Modeling and User Profiling: A Comprehensive Survey](https://mlsanity.com/paper/2402.09660/) (2024)
- [PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision](https://mlsanity.com/paper/2010.01494/) (2020)
- [AdaptSSR: Pre-training User Model with Augmentation-Adaptive Self-Supervised Ranking](https://mlsanity.com/paper/2310.09706/) (2023)

## Built on by (most cited first, 2 of 2)

- [HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs](https://mlsanity.com/paper/2601.19839/) (2026)
- [IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models](https://mlsanity.com/paper/2604.24002/) (2026)

## Abstract (arXiv)

This paper investigates the performance of multimodal pre-trained models in user profiling tasks based on visual-linguistic demographic data. These models are critical for adapting to the needs and preferences of human users in social robotics, thereby providing personalized responses and enhancing interaction quality. First, we introduce two datasets specifically curated to represent demographic characteristics derived from user facial images. Next, we evaluate the performance of a prominent contrastive multimodal pre-trained model, CLIP, on these datasets, both in its out-of-the-box state and after fine-tuning. Initial results indicate that CLIP performs suboptimal in matching images to demographic descriptions without fine-tuning. Although fine-tuning significantly enhances its predictive capacity, the model continues to exhibit limitations in effectively generalizing subtle demographic nuances. To address this, we propose adopting a masked image modeling strategy to improve generalization and better capture subtle demographic attributes. This approach offers a pathway for enhancing demographic sensitivity in multimodal user modeling tasks.