# Hardware-Aware Learned Representation Compression for Distributed In-Sensor Vision

arXiv: https://arxiv.org/abs/2609.13947 · HTML: https://mlsanity.com/paper/2609.13947/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 12 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Towards Image Understanding from Deep Compression without Decoding](https://mlsanity.com/paper/1803.06131/) (2018)
- [Visual Wake Words Dataset](https://mlsanity.com/paper/1906.05721/) (2019)
- [Real-Time Gaze Tracking with Event-Driven Eye Segmentation](https://mlsanity.com/paper/2201.07367/) (2022)
- [OpenEDS2020: Open Eyes Dataset](https://mlsanity.com/paper/2005.03876/) (2020)
- [P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications](https://mlsanity.com/paper/2203.04737/) (2022)
- [Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays](https://mlsanity.com/paper/2004.12525/) (2020)
- [Distributed On-Sensor Compute System for AR/VR Devices: A Semi-Analytical Simulation Framework for Power Estimation](https://mlsanity.com/paper/2203.07474/) (2022)
- [Sobol Sequence Optimization for Hardware-Efficient Vector Symbolic Architectures](https://mlsanity.com/paper/2311.10277/) (2023)
- [Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)](https://mlsanity.com/paper/2304.02968/) (2023)
- [OASIS: Optimized Lightweight Autoencoder System for Distributed In-Sensor computing](https://mlsanity.com/paper/2505.02256/) (2025)
- [Voltage-Controlled Magnetic Tunnel Junction based ADC-less Global Shutter Processing-in-Pixel for Extreme-Edge Intelligence](https://mlsanity.com/paper/2410.10592/) (2024)
- [AMS-HD: Hyperdimensional Computing for Real-Time and Energy-Efficient Acute Mountain Sickness Detection](https://mlsanity.com/paper/2602.08916/) (2026)

## Abstract (arXiv)

In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a 3×3×8 latent to a 64-dimensional hypervector provides an additional 1.77× communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall 18,816× reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately 2×-4.5× while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.