# Hand Gestures Recognition in Videos Taken with Lensless Camera

arXiv: https://arxiv.org/abs/2210.08233 · HTML: https://mlsanity.com/paper/2210.08233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 15 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Lensless computational imaging through deep learning](https://mlsanity.com/paper/1702.08516/) (2017)
- [Learned reconstructions for practical mask-based lensless imaging](https://mlsanity.com/paper/1908.11502/) (2019)
- [Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?](https://mlsanity.com/paper/2004.04968/) (2020)
- [ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI](https://mlsanity.com/paper/1705.06869/) (2017)
- [Deep speckle correlation: a deep learning approach towards scalable imaging through scattering media](https://mlsanity.com/paper/1806.04139/) (2018)
- [Privacy-Preserving Action Recognition using Coded Aperture Videos](https://mlsanity.com/paper/1902.09085/) (2019)
- [Reconstruction-free action inference from compressive imagers](https://mlsanity.com/paper/1501.04367/) (2015)
- [Direct Object Recognition Without Line-of-Sight Using Optical Coherence](https://mlsanity.com/paper/1903.07705/) (2019)
- [Lensless-camera based machine learning for image classification](https://mlsanity.com/paper/1709.00408/) (2017)
- [Text detection and recognition based on a lensless imaging system](https://mlsanity.com/paper/2210.04244/) (2022)

## Abstract (arXiv)

A lensless camera is an imaging system that uses a mask in place of a lens, making it thinner, lighter, and less expensive than a lensed camera. However, additional complex computation and time are required for image reconstruction. This work proposes a deep learning model named Raw3dNet that recognizes hand gestures directly on raw videos captured by a lensless camera without the need for image restoration. In addition to conserving computational resources, the reconstruction-free method provides privacy protection. Raw3dNet is a novel end-to-end deep neural network model for the recognition of hand gestures in lensless imaging systems. It is created specifically for raw video captured by a lensless camera and has the ability to properly extract and combine temporal and spatial features. The network is composed of two stages: 1. spatial feature extractor (SFE), which enhances the spatial features of each frame prior to temporal convolution; 2. 3D-ResNet, which implements spatial and temporal convolution of video streams. The proposed model achieves 98.59% accuracy on the Cambridge Hand Gesture dataset in the lensless optical experiment, which is comparable to the lensed-camera result. Additionally, the feasibility of physical object recognition is assessed. Furtherly, we show that the recognition can be achieved with respectable accuracy using only a tiny portion of the original raw data, indicating the potential for reducing data traffic in cloud computing scenarios.