# TinyVQA: Compact Multimodal Deep Neural Network for Visual Question Answering on Resource-Constrained Devices

arXiv: https://arxiv.org/abs/2404.03574 · HTML: https://mlsanity.com/paper/2404.03574/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding](https://mlsanity.com/paper/2012.02951/) (2020)
- [VQA-Aid: Visual Question Answering for Post-Disaster Damage Assessment and Analysis](https://mlsanity.com/paper/2106.10548/) (2021)
- [Squeezed Edge YOLO: Onboard Object Detection on Edge Devices](https://mlsanity.com/paper/2312.11716/) (2023)
- [ReProHRL: Towards Multi-Goal Navigation in the Real World using Hierarchical Agents](https://mlsanity.com/paper/2308.08737/) (2023)
- [HeteroEdge: Addressing Asymmetry in Heterogeneous Collaborative Autonomous Systems](https://mlsanity.com/paper/2305.03252/) (2023)
- [TinyM2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices](https://mlsanity.com/paper/2202.04303/) (2022)
- [Polar-VQA: Visual Question Answering on Remote Sensed Ice sheet Imagery from Polar Region](https://mlsanity.com/paper/2303.07403/) (2023)

## Built on by (most cited first, 5 of 5)

- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [From Tiny Machine Learning to Tiny Deep Learning: A Survey](https://mlsanity.com/paper/2506.18927/) (2025)
- [TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription](https://mlsanity.com/paper/2510.07098/) (2025)
- [Energy-Aware FPGA Implementation of Spiking Neural Network with LIF Neurons](https://mlsanity.com/paper/2411.01628/) (2024)
- [Damage Assessment after Natural Disasters with UAVs: Semantic Feature Extraction using Deep Learning](https://mlsanity.com/paper/2412.10756/) (2024)

## Abstract (arXiv)

Traditional machine learning models often require powerful hardware, making them unsuitable for deployment on resource-limited devices. Tiny Machine Learning (tinyML) has emerged as a promising approach for running machine learning models on these devices, but integrating multiple data modalities into tinyML models still remains a challenge due to increased complexity, latency, and power consumption. This paper proposes TinyVQA, a novel multimodal deep neural network for visual question answering tasks that can be deployed on resource-constrained tinyML hardware. TinyVQA leverages a supervised attention-based model to learn how to answer questions about images using both vision and language modalities. Distilled knowledge from the supervised attention-based VQA model trains the memory aware compact TinyVQA model and low bit-width quantization technique is employed to further compress the model for deployment on tinyML devices. The TinyVQA model was evaluated on the FloodNet dataset, which is used for post-disaster damage assessment. The compact model achieved an accuracy of 79.5%, demonstrating the effectiveness of TinyVQA for real-world applications. Additionally, the model was deployed on a Crazyflie 2.0 drone, equipped with an AI deck and GAP8 microprocessor. The TinyVQA model achieved low latencies of 56 ms and consumes 693 mW power while deployed on the tiny drone, showcasing its suitability for resource-constrained embedded systems.