# PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks

arXiv: https://arxiv.org/abs/2604.23148 · HTML: https://mlsanity.com/paper/2604.23148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 37 references that are in the map.
- First posted on arXiv 25 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling](https://mlsanity.com/paper/2410.16668/) (2024)
- [SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions](https://mlsanity.com/paper/2412.04036/) (2024)
- [Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory Analysis](https://mlsanity.com/paper/2406.01964/) (2024)
- [Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks](https://mlsanity.com/paper/2310.05595/) (2023)
- [Next-Generation Phishing: How LLM Agents Empower Cyber Attackers](https://mlsanity.com/paper/2411.13874/) (2024)
- [GazeNoter: Co-Piloted AR Note-Taking via Gaze Selection of LLM Suggestions to Match Users' Intentions](https://mlsanity.com/paper/2407.01161/) (2024)
- [On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks](https://mlsanity.com/paper/2504.13209/) (2025)
- [SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors](https://mlsanity.com/paper/2505.24458/) (2025)

## Built on by (most cited first, 1 of 1)

- [From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms](https://mlsanity.com/paper/2608.24877/) (2026)

## Abstract (arXiv)

The emerging threat of AR-LLM-based Social Engineering (AR-LLM-SE) attacks (e.g. SEAR) poses a significant risk to real-world social interactions. In such an attack, a malicious actor uses Augmented Reality (AR) glasses to capture a target visual and vocal data. A Large Language Model (LLM) then analyzes this data to identify the individual and generate a detailed social profile. Subsequently, LLM-powered agents employ social engineering strategies, providing real-time conversation suggestions, to gain the target trust and ultimately execute phishing or other malicious acts. Despite its potential, the practical application of AR-LLM-SE faces two major bottlenecks, (1) Cold-start personalization, Current Retrieval-Augmented Generation (RAG) methods introduce critical delays in the earliest turns, slowing initial profile formation and disrupting real-time interaction, (2) Static Attack Strategies, Existing approaches rely on fixed-stage, handcrafted social engineering tactics that lack foundation in established psychological theory. To address these limitations, we propose PhySE, a novel framework with two core innovations, (1) VLM-Based SocialContext Training, To eliminate profiling delays, we efficiently pre-train a Visual Language Model (VLM) with social-context data, enabling rapid, on-the-fly profile generation, (2) Adaptive Psychological Agent, We introduce a psychological LLM that dynamically deploys distinct classes of psychological strategies based on target response, moving beyond static, handcrafted scripts. We evaluated PhySE through an IRB-approved user study with 60 participants, collecting a novel dataset of 360 annotated conversations across diverse social scenarios.