# EyeCoD: Eye Tracking System Acceleration via FlatCam-based Algorithm & Accelerator Co-Design

arXiv: https://arxiv.org/abs/2206.00877 · HTML: https://mlsanity.com/paper/2206.00877/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2022.

## Built on by (most cited first, 6 of 6)

- [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022)
- [DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads](https://mlsanity.com/paper/2212.03414/) (2022)
- [SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators](https://mlsanity.com/paper/2405.00790/) (2024)
- [BOXR: Body and head motion Optimization framework for eXtended Reality](https://mlsanity.com/paper/2410.13084/) (2024)
- [FlatTrack: Eye-tracking with ultra-thin lensless cameras](https://mlsanity.com/paper/2501.15450/) (2025)
- [SteROI-D: System Design and Mapping for Stereo Depth Inference on Regions of Interest](https://mlsanity.com/paper/2502.09528/) (2025)

## Abstract (arXiv)

Eye tracking has become an essential human-machine interaction modality for providing immersive experience in numerous virtual and augmented reality (VR/AR) applications desiring high throughput (e.g., 240 FPS), small-form, and enhanced visual privacy. However, existing eye tracking systems are still limited by their: (1) large form-factor largely due to the adopted bulky lens-based cameras; and (2) high communication cost required between the camera and backend processor, thus prohibiting their more extensive applications. To this end, we propose a lensless FlatCam-based eye tracking algorithm and accelerator co-design framework dubbed EyeCoD to enable eye tracking systems with a much reduced form-factor and boosted system efficiency without sacrificing the tracking accuracy, paving the way for next-generation eye tracking solutions. On the system level, we advocate the use of lensless FlatCams to facilitate the small form-factor need in mobile eye tracking systems. On the algorithm level, EyeCoD integrates a predict-then-focus pipeline that first predicts the region-of-interest (ROI) via segmentation and then only focuses on the ROI parts to estimate gaze directions, greatly reducing redundant computations and data movements. On the hardware level, we further develop a dedicated accelerator that (1) integrates a novel workload orchestration between the aforementioned segmentation and gaze estimation models, (2) leverages intra-channel reuse opportunities for depth-wise layers, and (3) utilizes input feature-wise partition to save activation memory size. On-silicon measurement validates that our EyeCoD consistently reduces both the communication and computation costs, leading to an overall system speedup of 10.95x, 3.21x, and 12.85x over CPUs, GPUs, and a prior-art eye tracking processor called CIS-GEP, respectively, while maintaining the tracking accuracy.