# Kraken: A Direct Event/Frame-Based Multi-sensor Fusion SoC for Ultra-Efficient Visual Processing in Nano-UAVs

arXiv: https://arxiv.org/abs/2209.01065 · HTML: https://mlsanity.com/paper/2209.01065/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2022.

## Built on by (most cited first, 7 of 7)

- [A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms](https://mlsanity.com/paper/2306.15552/) (2023)
- [Neuromorphic Optical Flow and Real-time Implementation with Event Cameras](https://mlsanity.com/paper/2304.07139/) (2023)
- [ColibriUAV: An Ultra-Fast, Energy-Efficient Neuromorphic Edge Processing UAV-Platform with Event-Based and Frame-Based Cameras](https://mlsanity.com/paper/2305.18371/) (2023)
- [Ev-Edge: Efficient Execution of Event-based Vision Algorithms on Commodity Edge Platforms](https://mlsanity.com/paper/2403.15717/) (2024)
- [A Heterogeneous RISC-V based SoC for Secure Nano-UAV Navigation](https://mlsanity.com/paper/2401.03531/) (2024)
- [Advancing Neuromorphic Computing: Mixed-Signal Design Techniques Leveraging Brain Code Units and Fundamental Code Units](https://mlsanity.com/paper/2403.11563/) (2024)
- [Optimizing DNN Inference on Multi-Accelerator SoCs at Training-time](https://mlsanity.com/paper/2409.18566/) (2024)

## Abstract (arXiv)

Small-size unmanned aerial vehicles (UAV) have the potential to dramatically increase safety and reduce cost in applications like critical infrastructure maintenance and post-disaster search and rescue. Many scenarios require UAVs to shrink toward nano and pico-size form factors. The key open challenge to achieve true autonomy on Nano-UAVs is to run complex visual tasks like object detection, tracking, navigation and obstacle avoidance fully on board, at high speed and robustness, under tight payload and power constraints. With the Kraken SoC, fabricated in 22nm FDX technology, we demonstrate a multi-visual-sensor capability exploiting both event-based and BW/RGB imagers, combining their output for multi-functional visual tasks previously impossible on a single low-power chip for Nano-UAVs. Kraken is an ultra-low-power, heterogeneous SoC architecture integrating three acceleration engines and a vast set of peripherals to enable efficient interfacing with standard frame-based sensors and novel event-based DVS. Kraken enables highly sparse event-driven sub-uJ/inf SNN inference on a dedicated neuromorphic energy-proportional accelerator. Moreover, it can perform frame-based inference by combining a 1.8TOpsW 8-cores RISC-V processor cluster with mixed-precision DNN extensions with a 1036TOpsW TNN accelerator.