Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes, and dynamic tasks. We show that passive EEG, fused online with behavioral evidence, can meaningfully extend the number of targets users detect and engage beyond their unaided action bandwidth. OLIVE learns from both explicit behavioral signals (the targets the user shoots down in an XR first-person shooter game) and implicit physiological signals (fixation-locked EEG) to provide timely guidance, continuously adapting a frozen vision-language model's inference on which items are task-relevant by jointly estimating per-source reliability without manual labels or offline training. Through three user studies, including two live deployments of an assistive agent driven by OLIVE in XR, we show that OLIVE Pareto-dominates prior test-time adaptation frameworks, achieving the highest convergence rate at comparable convergence speed. Combining implicit physiological and explicit behavioral signals, the OLIVE agent produces the largest and most reliable within-session improvement to a user's ability to detect and engage targets, largely independent of the individual's skill. When the target switches silently, the agent that uses both behavioral and physiological signals reconverges significantly faster than the behavior-only agent (1.27 times faster on average, p = .008), restoring trustworthy guidance at the moment the task changes, precisely when reliable assistance matters most.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 95 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
13 of 95 references.
References · 13
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
- 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
- 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
- 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
- 2020Residual Policy Learning for Shared Autonomycited by 26
- 2021Adapting User Interfaces with Model-based Reinforcement Learningcited by 17
- 2023How does HCI Understand Human Autonomy and Agency?cited by 14
- 2016Neural mechanisms underlying catastrophic failure in human-machine interaction during aerial navigationcited by 2
- 2025Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenescited by 2
- 2024Fine-tuning Myoelectric Control through Reinforcement Learning in a Game Environmentcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Accelerating Reinforcement Learning via Error-Related Human Brain Signals
- 2026AutoBCI: Forecast-Guided Agentic Neural Architecture Discovery for EEG-Based Brain--Computer Interfaces
- 2020Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback
- 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
- 2026Multi-Modal Multi-Agent Robotic Cognitive Alignment enabled by Non-Invasive Consumer Brain Computer Interfaces: A Proof of Concept Exploration
- 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
- 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
- 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback