# HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs

arXiv: https://arxiv.org/abs/2601.19839 · HTML: https://mlsanity.com/paper/2601.19839/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents](https://mlsanity.com/paper/2308.07241/) (2023)
- [Facial Geometric Detail Recovery via Implicit Representation](https://mlsanity.com/paper/2203.09692/) (2022)
- [Personalized Dialogue Generation with Persona-Adaptive Attention](https://mlsanity.com/paper/2210.15088/) (2022)
- [Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2405.17974/) (2024)
- [PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization](https://mlsanity.com/paper/2506.12915/) (2025)
- [Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots](https://mlsanity.com/paper/2103.09534/) (2021)
- [MIRACLE: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control](https://mlsanity.com/paper/2310.18342/) (2023)
- ["In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning](https://mlsanity.com/paper/2403.03102/) (2024)
- [Building a Personalized Dialogue System with Prompt-Tuning](https://mlsanity.com/paper/2206.05399/) (2022)
- [USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions](https://mlsanity.com/paper/2502.10636/) (2025)
- [Reasoning LLMs for User-Aware Multimodal Conversational Agents](https://mlsanity.com/paper/2504.01700/) (2025)
- [Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users](https://mlsanity.com/paper/2310.20479/) (2023)
- [Interruption Handling for Conversational Robots](https://mlsanity.com/paper/2501.01568/) (2025)
- [Demographic User Modeling for Social Robotics with Multimodal Pre-trained Models](https://mlsanity.com/paper/2502.10642/) (2025)
- [On the Impact of Interruptions During Multi-Robot Supervision Tasks](https://mlsanity.com/paper/2306.16501/) (2023)

## Built on by (most cited first, 1 of 1)

- [IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models](https://mlsanity.com/paper/2604.24002/) (2026)

## Abstract (arXiv)

Existing human-robot interaction systems often lack mechanisms for sustained personalization and dynamic adaptation in multi-user environments, limiting their effectiveness in real-world deployments. We present HARMONI, a multimodal personalization framework that leverages large language models to enable socially assistive robots to manage long-term multi-user interactions. The framework integrates four key modules: (i) a perception module that identifies active speakers and extracts multimodal input; (ii) a world modeling module that maintains representations of the environment and short-term conversational context; (iii) a user modeling module that updates long-term speaker-specific profiles; and (iv) a generation module that produces contextually grounded and ethically informed responses. Through extensive evaluation and ablation studies on four datasets, as well as a real-world scenario-driven user-study in a nursing home environment, we demonstrate that HARMONI supports robust speaker identification, online memory updating, and ethically aligned personalization, outperforming baseline LLM-driven approaches in user modeling accuracy, personalization quality, and user satisfaction.