Artificial intelligence optical hardware empowers high-resolution hyperspectral video understanding at 1.2 Tb/s
Not on the map. Why?
Abstract (arXiv)
Foundation models, exemplified by GPT technology, are discovering new horizons in artificial intelligence by executing tasks beyond their designers' expectations. While the present generation provides fundamental advances in understanding language and images, the next frontier is video comprehension. Progress in this area must overcome the 1 Tb/s data rate demanded to grasp real-time multidimensional video information. This speed limit lies well beyond the capabilities of the existing generation of hardware, imposing a roadblock to further advances. This work introduces a hardware-accelerated integrated optoelectronic platform for multidimensional video understanding in real-time. The technology platform combines artificial intelligence hardware, processing information optically, with state-of-the-art machine vision networks, resulting in a data processing speed of 1.2 Tb/s with hundreds of frequency bands and megapixel spatial resolution at video rates. Such performance, validated in the AI tasks of video semantic segmentation and object understanding in indoor and aerial applications, surpasses the speed of the closest technologies with similar spectral resolution by three to four orders of magnitude. This platform opens up new avenues for research in real-time AI video understanding of multidimensional visual information, helping the empowerment of future human-machine interactions and cognitive processing developments.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 54 references that are in the map. First posted on arXiv 17 Dec 2023.
Builds on
12 of 54 references.
References · 12
- 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
- 2022Learning Video Representations from Large Language Modelscited by 204
- 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
- 2022Revisiting the "Video" in Video-Language Understandingcited by 167
- 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
- 2021FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendationcited by 97
- 2022Unified Transformer Tracker for Object Trackingcited by 31
- 2022Snapshot Multispectral Imaging Using a Diffractive Optical Networkcited by 16
- 2020Human action recognition with a large-scale brain-inspired photonic computercited by 15
- 2020Complex Sequential Understanding through the Awareness of Spatial and Temporal Conceptscited by 8
- 2022Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoderscited by 6
Built on by
Not cited yet.