# FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence

arXiv: https://arxiv.org/abs/2605.22868 · HTML: https://mlsanity.com/paper/2605.22868/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 20 of 20)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detection](https://mlsanity.com/paper/2303.17895/) (2023)
- [Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression](https://mlsanity.com/paper/2004.10043/) (2020)
- [SynDrone -- Multi-modal UAV Dataset for Urban Scenarios](https://mlsanity.com/paper/2308.10491/) (2023)
- [Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation](https://mlsanity.com/paper/1705.10422/) (2017)
- [Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors](https://mlsanity.com/paper/2012.00201/) (2020)
- [MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation](https://mlsanity.com/paper/2406.18815/) (2024)
- [Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series](https://mlsanity.com/paper/1905.13570/) (2019)
- [Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images](https://mlsanity.com/paper/1803.03435/) (2018)
- [HyperSense: Hyperdimensional Intelligent Sensing for Energy-Efficient Sparse Data Processing](https://mlsanity.com/paper/2401.10267/) (2024)
- [UFO: Uncertainty-aware LiDAR-image Fusion for Off-road Semantic Terrain Map Estimation](https://mlsanity.com/paper/2403.02642/) (2024)
- [A Plug-in Tiny AI Module for Intelligent and Selective Sensor Data Transmission](https://mlsanity.com/paper/2402.02043/) (2024)
- [Hyperdimensional Intelligent Sensing for Efficient Real-Time Audio Processing on Extreme Edge](https://mlsanity.com/paper/2502.10718/) (2025)
- [DecoHD: Decomposed Hyperdimensional Classification under Extreme Memory Budgets](https://mlsanity.com/paper/2511.03911/) (2025)
- [LogHD: Robust Compression of Hyperdimensional Classifiers via Logarithmic Class-Axis Reduction](https://mlsanity.com/paper/2511.03938/) (2025)

## Abstract (arXiv)

Autonomous systems and smart-industry deployments increasingly split computation across near-sensor, edge, and cloud resources, where tight energy, latency, and reliability budgets demand run-time adaptivity. In practice, deciding what to compute and transmit at each point is pivotal; yet as multimodal sensor suites (cameras, LiDAR/depth, etc.) proliferate at the edge, most prior approaches either (i) fuse modalities on powerful servers or (ii) apply uni-modal near-sensor filters that ignore cross-modal dependencies, leading to redundant transmissions or missed events. We present FusionSense, a fusion-aware intelligent sensing framework for energy-constrained autonomous edge systems. Lightweight near-sensor classifiers are trained via a three-step procedure: (i) a server-side fusion model learns the downstream task, (ii) filter-out-safe (FoS) labels quantify each modality's necessity relative to the fused decision, and (iii) an edge-side fusion model is compacted by injecting near-sensor predictions as auxiliary signals. The result is a run-time decision layer that jointly reduces compute and communication while scaling linearly with sensor count. On a dual-modality (RGB+Depth/LiDAR) setup with SynDrone, FusionSense sustains task quality at substantially higher data-reduction rates than uni-modal filters and delivers large end-to-end gains: up to 33x lower energy at 1% FoI prevalence, 11x at 10%, a 92.3% reduction in quality loss at a fixed 30% data reduction, and roughly 1.5x higher energy savings than the best prior filtering baseline.