# Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence

arXiv: https://arxiv.org/abs/2608.30369 · HTML: https://mlsanity.com/paper/2608.30369/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 95 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [Adaptive Policy Learning for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2303.07693/) (2023)
- [Frustratingly Easy Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18330/) (2024)
- [Residual Policy Learning for Shared Autonomy](https://mlsanity.com/paper/2004.05097/) (2020)
- [Adapting User Interfaces with Model-based Reinforcement Learning](https://mlsanity.com/paper/2103.06807/) (2021)
- [How does HCI Understand Human Autonomy and Agency?](https://mlsanity.com/paper/2301.12490/) (2023)
- [Neural mechanisms underlying catastrophic failure in human-machine interaction during aerial navigation](https://mlsanity.com/paper/1605.01784/) (2016)
- [Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes](https://mlsanity.com/paper/2508.01853/) (2025)
- [Fine-tuning Myoelectric Control through Reinforcement Learning in a Game Environment](https://mlsanity.com/paper/2411.13327/) (2024)
- [Error-related Potential driven Reinforcement Learning for adaptive Brain-Computer Interfaces](https://mlsanity.com/paper/2502.18594/) (2025)

## Abstract (arXiv)

We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes, and dynamic tasks. We show that passive EEG, fused online with behavioral evidence, can meaningfully extend the number of targets users detect and engage beyond their unaided action bandwidth. OLIVE learns from both explicit behavioral signals (the targets the user shoots down in an XR first-person shooter game) and implicit physiological signals (fixation-locked EEG) to provide timely guidance, continuously adapting a frozen vision-language model's inference on which items are task-relevant by jointly estimating per-source reliability without manual labels or offline training. Through three user studies, including two live deployments of an assistive agent driven by OLIVE in XR, we show that OLIVE Pareto-dominates prior test-time adaptation frameworks, achieving the highest convergence rate at comparable convergence speed. Combining implicit physiological and explicit behavioral signals, the OLIVE agent produces the largest and most reliable within-session improvement to a user's ability to detect and engage targets, largely independent of the individual's skill. When the target switches silently, the agent that uses both behavioral and physiological signals reconverges significantly faster than the behavior-only agent (1.27 times faster on average, p = .008), restoring trustworthy guidance at the moment the task changes, precisely when reliable assistance matters most.