# SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors

arXiv: https://arxiv.org/abs/2505.24458 · HTML: https://mlsanity.com/paper/2505.24458/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 4 of its 13 references that are in the map.
- First posted on arXiv 30 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 4 of 4)

- [SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words](https://mlsanity.com/paper/2406.13340/) (2024)
- [Detecting and Characterizing Lateral Phishing at Scale](https://mlsanity.com/paper/1910.00790/) (2019)
- [Breaching the Human Firewall: Social engineering in Phishing and Spear-Phishing Emails](https://mlsanity.com/paper/1606.00887/) (2016)
- [Privacy-preserving Reflection Rendering for Augmented Reality](https://mlsanity.com/paper/2207.03056/) (2022)

## Built on by (most cited first, 2 of 2)

- [UNSEEN: A Cross-Stack LLM Unlearning Defense against AR-LLM Social Engineering Attacks](https://mlsanity.com/paper/2604.23141/) (2026)
- [PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks](https://mlsanity.com/paper/2604.23148/) (2026)

## Abstract (arXiv)

The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.