# Helios: An extremely low power event-based gesture recognition for always-on smart eyewear

arXiv: https://arxiv.org/abs/2407.05206 · HTML: https://mlsanity.com/paper/2407.05206/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Graph-Based Object Classification for Neuromorphic Vision Sensing](https://mlsanity.com/paper/1908.06648/) (2019)
- [From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection](https://mlsanity.com/paper/2304.13455/) (2023)
- [EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream](https://mlsanity.com/paper/2012.06475/) (2020)
- [Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach](https://mlsanity.com/paper/2404.00834/) (2024)
- [Hierarchical Neural Memory Network for Low Latency Event Processing](https://mlsanity.com/paper/2305.17852/) (2023)
- [Scene Adaptive Sparse Transformer for Event-based Object Detection](https://mlsanity.com/paper/2404.01882/) (2024)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams](https://mlsanity.com/paper/2404.08640/) (2024)
- [EventPoint: Self-Supervised Interest Point Detection and Description for Event-based Camera](https://mlsanity.com/paper/2109.00210/) (2021)
- [LEOD: Label-Efficient Object Detection for Event Cameras](https://mlsanity.com/paper/2311.17286/) (2023)
- [A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization](https://mlsanity.com/paper/2403.19412/) (2024)
- [SEVD: Synthetic Event-based Vision Dataset for Ego and Fixed Traffic Perception](https://mlsanity.com/paper/2404.10540/) (2024)
- [Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN](https://mlsanity.com/paper/2404.14279/) (2024)

## Built on by (most cited first, 8 of 8)

- [EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera](https://mlsanity.com/paper/2503.12419/) (2025)
- [Reading in the Dark with Foveated Event Vision](https://mlsanity.com/paper/2506.06918/) (2025)
- [Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision](https://mlsanity.com/paper/2603.26646/) (2026)
- [EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras](https://mlsanity.com/paper/2605.12297/) (2026)
- [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025)
- [OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision](https://mlsanity.com/paper/2606.07431/) (2026)
- [From Muscle to Text with MyoText: sEMG to Text via Finger Classification and Transformer-Based Decoding](https://mlsanity.com/paper/2601.03098/) (2026)
- [AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization](https://mlsanity.com/paper/2609.19793/) (2026)

## Abstract (arXiv)

This paper introduces Helios, the first extremely low-power, real-time, event-based hand gesture recognition system designed for all-day on smart eyewear. As augmented reality (AR) evolves, current smart glasses like the Meta Ray-Bans prioritize visual and wearable comfort at the expense of functionality. Existing human-machine interfaces (HMIs) in these devices, such as capacitive touch and voice controls, present limitations in ergonomics, privacy and power consumption. Helios addresses these challenges by leveraging natural hand interactions for a more intuitive and comfortable user experience. Our system utilizes a extremely low-power and compact 3mmx4mm/20mW event camera to perform natural hand-based gesture recognition for always-on smart eyewear. The camera's output is processed by a convolutional neural network (CNN) running on a NXP Nano UltraLite compute platform, consuming less than 350mW. Helios can recognize seven classes of gestures, including subtle microgestures like swipes and pinches, with 91% accuracy. We also demonstrate real-time performance across 20 users at a remarkably low latency of 60ms. Our user testing results align with the positive feedback we received during our recent successful demo at AWE-USA-2024.