# Privacy-Enhancing Optical Embeddings for Lensless Classification

arXiv: https://arxiv.org/abs/2211.12864 · HTML: https://mlsanity.com/paper/2211.12864/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [The Limitations of Adversarial Training and the Blind-Spot Attack](https://mlsanity.com/paper/1901.04684/) (2019)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Privacy-Preserving Human Activity Recognition from Extreme Low Resolution](https://mlsanity.com/paper/1604.03196/) (2016)
- [Learning Privacy Preserving Encodings through Adversarial Training](https://mlsanity.com/paper/1802.05214/) (2018)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [Privacy-Preserving Action Recognition using Coded Aperture Videos](https://mlsanity.com/paper/1902.09085/) (2019)
- [Random Projections through multiple optical scattering: Approximating kernels at the speed of light](https://mlsanity.com/paper/1510.06664/) (2015)
- [Lensless Imaging by Compressive Sensing](https://mlsanity.com/paper/1305.7181/) (2013)
- [LenslessPiCam: A Hardware and Software Platform for Lensless Computational Imaging with a Raspberry Pi](https://mlsanity.com/paper/2206.01430/) (2022)
- [Known-plaintext attack and ciphertext-only attack for encrypted single-pixel imaging](https://mlsanity.com/paper/1905.13594/) (2019)
- [pyFFS: A Python Library for Fast Fourier Series Computation and Interpolation with GPU Acceleration](https://mlsanity.com/paper/2110.00262/) (2021)

## Built on by (most cited first, 3 of 3)

- [Nonlinear optical encoding enabled by recurrent linear scattering](https://mlsanity.com/paper/2307.08558/) (2023)
- [PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging](https://mlsanity.com/paper/2409.17996/) (2024)
- [LenslessFace: An End-to-End Optimized Lensless System for Privacy-Preserving Face Verification](https://mlsanity.com/paper/2406.04129/) (2024)

## Abstract (arXiv)

Lensless imaging can provide visual privacy due to the highly multiplexed characteristic of its measurements. However, this alone is a weak form of security, as various adversarial attacks can be designed to invert the one-to-many scene mapping of such cameras. In this work, we enhance the privacy provided by lensless imaging by (1) downsampling at the sensor and (2) using a programmable mask with variable patterns as our optical encoder. We build a prototype from a low-cost LCD and Raspberry Pi components, for a total cost of around 100 USD. This very low price point allows our system to be deployed and leveraged in a broad range of applications. In our experiments, we first demonstrate the viability and reconfigurability of our system by applying it to various classification tasks: MNIST, CelebA (face attributes), and CIFAR10. By jointly optimizing the mask pattern and a digital classifier in an end-to-end fashion, low-dimensional, privacy-enhancing embeddings are learned directly at the sensor. Secondly, we show how the proposed system, through variable mask patterns, can thwart adversaries that attempt to invert the system (1) via plaintext attacks or (2) in the event of camera parameters leaks. We demonstrate the defense of our system to both risks, with 55% and 26% drops in image quality metrics for attacks based on model-based convex optimization and generative neural networks respectively. We open-source a wave propagation and camera simulator needed for end-to-end optimization, the training software, and a library for interfacing with the camera.