# MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding

arXiv: https://arxiv.org/abs/2510.25327 · HTML: https://mlsanity.com/paper/2510.25327/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [PyPose: A Library for Robot Learning with Physics-based Optimization](https://mlsanity.com/paper/2209.15428/) (2022)
- [It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation](https://mlsanity.com/paper/2102.01393/) (2021)
- [ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles](https://mlsanity.com/paper/2010.10754/) (2020)
- [ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer's Disease](https://mlsanity.com/paper/2310.15301/) (2023)
- [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://mlsanity.com/paper/2410.01270/) (2024)
- [AdaFlow: Opportunistic Inference on Asynchronous Mobile Data with Generalized Affinity Control](https://mlsanity.com/paper/2410.24028/) (2024)

## Built on by (most cited first, 3 of 3)

- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [RETRACE: Resilience-Guided Trait-Conditioned Craving Estimation from Wearable Physiology in Opioid Use Disorder](https://mlsanity.com/paper/2608.14947/) (2026)
- [MM-BEV: Enhancing Timeliness by Computing Where and When it Matters](https://mlsanity.com/paper/2608.15437/) (2026)

## Abstract (arXiv)

Real-time multimodal inference on resource-constrained edge devices is essential for applications such as autonomous driving, human-computer interaction, and mobile health. However, prior work often overlooks the tight coupling between sensing dynamics and model execution, as well as the complex inter-modality dependencies. In this paper, we propose MMEdge, a new on-device multimodal inference framework based on pipelined sensing and encoding. Instead of waiting for complete sensor inputs, MMEdge decomposes the entire inference process into a sequence of fine-grained sensing and encoding units, allowing computation to proceed incrementally as data arrive. MMEdge also introduces a lightweight but effective temporal aggregation module that captures rich temporal dynamics across different pipelined units to maintain accuracy performance. Such pipelined design also opens up opportunities for fine-grained cross-modal optimization and early decision-making during inference. To further enhance system performance under resource variability and input data complexity, MMEdge incorporates an adaptive multimodal configuration optimizer that dynamically selects optimal sensing and model configurations for each modality under latency constraints, and a cross-modal speculative skipping mechanism that bypasses future units of slower modalities when early predictions reach sufficient confidence. We evaluate MMEdge using two public multimodal datasets and deploy it on a real-world unmanned aerial vehicle (UAV)-based multimodal testbed. The results show that MMEdge significantly reduces end-to-end latency while maintaining high task accuracy across various system and data dynamics. A video demonstration of MMEdge's performance in real world is available at https://youtu.be/qRew7sT-iWw.