# CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras

arXiv: https://arxiv.org/abs/2406.09409 · HTML: https://mlsanity.com/paper/2406.09409/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 74 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learning](https://mlsanity.com/paper/2003.13446/) (2020)
- [Event Based, Near Eye Gaze Tracking Beyond 10,000Hz](https://mlsanity.com/paper/2004.03577/) (2020)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras](https://mlsanity.com/paper/2002.11656/) (2020)
- [EVEN: An Event-Based Framework for Monocular Depth Estimation at Adverse Night Conditions](https://mlsanity.com/paper/2302.03860/) (2023)
- [Event Camera Based Real-Time Detection and Tracking of Indoor Ground Robots](https://mlsanity.com/paper/2102.11916/) (2021)
- [Towards Interpretable Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/2108.05312/) (2021)
- [Fourier DiffuserScope: Single-shot 3D Fourier light field microscopy with a diffuser](https://mlsanity.com/paper/2006.16343/) (2020)
- [TiDy-PSFs: Computational Imaging with Time-Averaged Dynamic Point-Spread-Functions](https://mlsanity.com/paper/2303.17583/) (2023)
- [Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight](https://mlsanity.com/paper/2310.09875/) (2023)
- [PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing](https://mlsanity.com/paper/2207.00945/) (2022)

## Built on by (most cited first, 7 of 7)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation](https://mlsanity.com/paper/2412.07761/) (2024)
- [Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light](https://mlsanity.com/paper/2509.06741/) (2025)
- [EventTracer: Fast Path Tracing-based Event Stream Rendering](https://mlsanity.com/paper/2508.18071/) (2025)
- [Optical Linear Systems Framework for Event Sensing and Computational Neuromorphic Imaging](https://mlsanity.com/paper/2601.13498/) (2026)
- [Programmable Silicon Retina on Pixel Processor Array](https://mlsanity.com/paper/2606.08370/) (2026)
- [Modelling dynamic systems transfer functions from events in computational neuromorphic imaging](https://mlsanity.com/paper/2609.29863/) (2026)

## Abstract (arXiv)

Point-spread-function (PSF) engineering is a well-established computational imaging technique that uses phase masks and other optical elements to embed extra information (e.g., depth) into the images captured by conventional CMOS image sensors. To date, however, PSF-engineering has not been applied to neuromorphic event cameras; a powerful new image sensing technology that responds to changes in the log-intensity of light. This paper establishes theoretical limits (Cram'er Rao bounds) on 3D point localization and tracking with PSF-engineered event cameras. Using these bounds, we first demonstrate that existing Fisher phase masks are already near-optimal for localizing static flashing point sources (e.g., blinking fluorescent molecules). We then demonstrate that existing designs are sub-optimal for tracking moving point sources and proceed to use our theory to design optimal phase masks and binary amplitude masks for this task. To overcome the non-convexity of the design problem, we leverage novel implicit neural representation based parameterizations of the phase and amplitude masks. We demonstrate the efficacy of our designs through extensive simulations. We also validate our method with a simple prototype.