# SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions

arXiv: https://arxiv.org/abs/2412.04036 · HTML: https://mlsanity.com/paper/2412.04036/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 78 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis](https://mlsanity.com/paper/2402.05863/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [Towards Human-centered Proactive Conversational Agents](https://mlsanity.com/paper/2404.12670/) (2024)
- [Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services](https://mlsanity.com/paper/2404.16283/) (2024)
- [Let's Negotiate! A Survey of Negotiation Dialogue Systems](https://mlsanity.com/paper/2212.09072/) (2022)
- [Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization](https://mlsanity.com/paper/2201.01928/) (2022)
- [DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge](https://mlsanity.com/paper/2405.12541/) (2024)
- [Towards Multimodal Emotional Support Conversation Systems](https://mlsanity.com/paper/2408.03650/) (2024)
- [SocialDial: A Benchmark for Socially-Aware Dialogue Systems](https://mlsanity.com/paper/2304.12026/) (2023)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [Exploring and Characterizing Large Language Models For Embedded System Development and Debugging](https://mlsanity.com/paper/2307.03817/) (2023)
- [Assistive Large Language Model Agents for Socially-Aware Negotiation Dialogues](https://mlsanity.com/paper/2402.01737/) (2024)
- [Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time](https://mlsanity.com/paper/2406.09569/) (2024)
- [RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations](https://mlsanity.com/paper/2402.11178/) (2024)
- ["It's the only thing I can trust": Envisioning Large Language Model Use by Autistic Workers for Communication Assistance](https://mlsanity.com/paper/2403.03297/) (2024)
- [GesturePrint: Enabling User Identification for mmWave-based Gesture Recognition Systems](https://mlsanity.com/paper/2408.05358/) (2024)
- [Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation](https://mlsanity.com/paper/2409.09135/) (2024)
- [SADAS: A Dialogue Assistant System Towards Remediating Norm Violations in Bilingual Socio-Cultural Conversations](https://mlsanity.com/paper/2402.01736/) (2024)

## Built on by (most cited first, 25 of 25)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions](https://mlsanity.com/paper/2505.14668/) (2025)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents](https://mlsanity.com/paper/2508.20973/) (2025)
- [An Egocentric Vision-Language Model based Portable Real-time Smart Assistant](https://mlsanity.com/paper/2503.04250/) (2025)
- [Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding](https://mlsanity.com/paper/2504.02878/) (2025)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Tasks](https://mlsanity.com/paper/2605.04227/) (2026)
- [On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks](https://mlsanity.com/paper/2504.13209/) (2025)
- [Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies](https://mlsanity.com/paper/2505.09938/) (2025)
- [DiscussLLM: Teaching Large Language Models When to Speak](https://mlsanity.com/paper/2508.18167/) (2025)
- [EgoSocial: Benchmarking Proactive Intervention Ability of Omnimodal LLMs via Egocentric Social Interaction Perception](https://mlsanity.com/paper/2510.13105/) (2025)
- [Adaptive Human-Agent Teaming: A Review of Empirical Studies from the Process Dynamics Perspective](https://mlsanity.com/paper/2504.10918/) (2025)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [SoulSeek: Exploring the Use of Social Cues in LLM-based Information Seeking](https://mlsanity.com/paper/2601.01094/) (2026)
- [SensorPersona: An LLM-Empowered System for Continual Persona Extraction from Longitudinal Mobile Sensor Streams](https://mlsanity.com/paper/2604.06204/) (2026)
- ["Excuse me, may I say something..." CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborations](https://mlsanity.com/paper/2604.15588/) (2026)
- [UNSEEN: A Cross-Stack LLM Unlearning Defense against AR-LLM Social Engineering Attacks](https://mlsanity.com/paper/2604.23141/) (2026)
- [PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks](https://mlsanity.com/paper/2604.23148/) (2026)
- [Agents in the Large: Perception-Centered Architecture for Persistent Agents](https://mlsanity.com/paper/2608.30478/) (2026)
- [After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants From Large-Scale Persona Interactions](https://mlsanity.com/paper/2602.04000/) (2026)
- [Large language models for spreading dynamics in complex systems](https://mlsanity.com/paper/2602.08085/) (2026)
- [SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VR](https://mlsanity.com/paper/2603.00522/) (2026)
- [Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions](https://mlsanity.com/paper/2608.14132/) (2026)

## Abstract (arXiv)

Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.