# Time-multiplexed Neural Holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators

arXiv: https://arxiv.org/abs/2205.02367 · HTML: https://mlsanity.com/paper/2205.02367/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 57 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 8 of 8)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [Learned holographic light transport](https://mlsanity.com/paper/2108.08253/) (2021)
- [High-contrast, speckle-free, true 3D holography via binary CGH optimization](https://mlsanity.com/paper/2201.02619/) (2022)

## Built on by (most cited first, 14 of 14)

- [Optical Generative Models](https://mlsanity.com/paper/2410.17970/) (2024)
- [TiDy-PSFs: Computational Imaging with Time-Averaged Dynamic Point-Spread-Functions](https://mlsanity.com/paper/2303.17583/) (2023)
- [Realistic Defocus Blur for Multiplane Computer-Generated Holography](https://mlsanity.com/paper/2205.07030/) (2022)
- [Stochastic Light Field Holography](https://mlsanity.com/paper/2307.06277/) (2023)
- [Maximum Likelihood Reconstruction for Multi-Look Digital Holography with Markov-Modeled Speckle Correlation](https://mlsanity.com/paper/2604.20154/) (2026)
- [High-performance real-world optical computing trained by in situ gradient-based model-free optimization](https://mlsanity.com/paper/2307.11957/) (2023)
- [Configurable Holography: Towards Display and Scene Adaptation](https://mlsanity.com/paper/2405.01558/) (2024)
- [HoloChrome: Polychromatic Illumination for Speckle Reduction in Holographic Near-Eye Displays](https://mlsanity.com/paper/2410.24144/) (2024)
- [3D Skew Gaussian Splatting with Any Camera Trajectory Visualization Engine](https://mlsanity.com/paper/2605.18334/) (2026)
- [Neural Lithography: Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Photolithography Simulator](https://mlsanity.com/paper/2309.17343/) (2023)
- [Large-Area Fabrication-Aware Computational Diffractive Optics](https://mlsanity.com/paper/2505.22313/) (2025)
- [Super-resolution image projection over an extended depth of field using a diffractive decoder](https://mlsanity.com/paper/2510.03938/) (2025)
- [Complex-Valued 2D Gaussian Representation for Computer-Generated Holography](https://mlsanity.com/paper/2511.15022/) (2025)
- [JPEG-Inspired Cloud-Edge Holography](https://mlsanity.com/paper/2512.12367/) (2025)

## Abstract (arXiv)

Holographic near-eye displays offer unprecedented capabilities for virtual and augmented reality systems, including perceptually important focus cues. Although artificial intelligence--driven algorithms for computer-generated holography (CGH) have recently made much progress in improving the image quality and synthesis efficiency of holograms, these algorithms are not directly applicable to emerging phase-only spatial light modulators (SLM) that are extremely fast but offer phase control with very limited precision. The speed of these SLMs offers time multiplexing capabilities, essentially enabling partially-coherent holographic display modes. Here we report advances in camera-calibrated wave propagation models for these types of holographic near-eye displays and we develop a CGH framework that robustly optimizes the heavily quantized phase patterns of fast SLMs. Our framework is flexible in supporting runtime supervision with different types of content, including 2D and 2.5D RGBD images, 3D focal stacks, and 4D light fields. Using our framework, we demonstrate state-of-the-art results for all of these scenarios in simulation and experiment.