A Framework for Event-based Computer Vision on a Mobile Device
Not on the map. Why?
Abstract (arXiv)
We present the first publicly available Android framework to stream data from an event camera directly to a mobile phone. Today's mobile devices handle a wider range of workloads than ever before and they incorporate a growing gamut of sensors that make devices smarter, more user friendly and secure. Conventional cameras in particular play a central role in such tasks, but they cannot record continuously, as the amount of redundant information recorded is costly to process. Bio-inspired event cameras on the other hand only record changes in a visual scene and have shown promising low-power applications that specifically suit mobile tasks such as face detection, gesture recognition or gaze tracking. Our prototype device is the first step towards embedding such an event camera into a battery-powered handheld device. The mobile framework allows us to stream events in real-time and opens up the possibilities for always-on and on-demand sensing on mobile phones. To liaise the asynchronous event camera output with synchronous von Neumann hardware, we look at how buffering events and processing them in batches can benefit mobile applications. We evaluate our framework in terms of latency and throughput and show examples of computer vision tasks that involve both event-by-event and pre-trained neural network methods for gesture recognition, aperture robust optical flow and grey-level image reconstruction from events. The code is available at https://github.com/neuromorphic-paris/frog
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 32 references that are in the map. First posted on arXiv 13 May 2022.
Builds on
12 of 32 references.
References · 12
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2018HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classificationcited by 218
- 2019End-to-End Learning of Representations for Asynchronous Event-Based Datacited by 217
- 2017Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarioscited by 150
- 2018Bringing a Blurry Frame Alive at High Frame-Rate with an Event Cameracited by 141
- 2019On-Device Neural Net Inference with Mobile GPUscited by 32
- 2018Real-time high speed motion prediction using fast aperture-robust event-driven visual flowcited by 24
- 2018Event-based Face Detection and Tracking in the Blink of an Eyecited by 16
- 2018Event-based Gesture Recognition with Dynamic Background Suppression using Smartphone Computational Capabilitiescited by 13
- 2018A Sparse Coding Multi-Scale Precise-Timing Machine Learning Algorithm for Neuromorphic Event-Based Sensorscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.