# Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses

arXiv: https://arxiv.org/abs/2306.13104 · HTML: https://mlsanity.com/paper/2306.13104/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 65 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Multi-dueling Bandits with Dependent Arms](https://mlsanity.com/paper/1705.00253/) (2017)
- [Preferential Bayesian optimisation with Skew Gaussian Processes](https://mlsanity.com/paper/2008.06677/) (2020)
- [Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses](https://mlsanity.com/paper/2205.13623/) (2022)
- [Preference-Based Learning for User-Guided HZD Gait Generation on Bipedal Walking Robots](https://mlsanity.com/paper/2011.05424/) (2020)
- [Efficient Exploration in Binary and Preferential Bayesian Optimization](https://mlsanity.com/paper/2110.09361/) (2021)
- [A Neural-Network Framework for the Design of Individualised Hearing-Loss Compensation](https://mlsanity.com/paper/2207.07091/) (2022)
- [Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses](https://mlsanity.com/paper/2209.13561/) (2022)

## Built on by (most cited first, 8 of 8)

- [Optimizing Retinal Prosthetic Stimuli with Conditional Invertible Neural Networks](https://mlsanity.com/paper/2403.04884/) (2024)
- [Evaluating Deep Human-in-the-Loop Optimization for Retinal Implants Using Sighted Participants](https://mlsanity.com/paper/2502.00177/) (2025)
- [Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models](https://mlsanity.com/paper/2509.00787/) (2025)
- [Visual Fixation-Based Retinal Prosthetic Simulation](https://mlsanity.com/paper/2410.11688/) (2024)
- [Regularized GLISp for sensor-guided human-in-the-loop optimization](https://mlsanity.com/paper/2511.04751/) (2025)
- [Fuzzing the brain: Automated stress testing for the safety of ML-driven neurostimulation](https://mlsanity.com/paper/2512.05383/) (2025)
- [SymbolSight: Minimizing Inter-Symbol Interference for Reading with Prosthetic Vision](https://mlsanity.com/paper/2601.17326/) (2026)
- [Percept-Aware Surgical Planning for Visual Cortical Prostheses with Vascular Avoidance](https://mlsanity.com/paper/2603.00362/) (2026)

## Abstract (arXiv)

Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to optimize patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. We show that our approach quickly learns a personalized stimulus encoder, leads to dramatic improvements in the quality of restored vision, and is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.