# Multimodal Remote Inference

arXiv: https://arxiv.org/abs/2508.07555 · HTML: https://mlsanity.com/paper/2508.07555/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 77 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Age of Information: An Introduction and Survey](https://mlsanity.com/paper/2007.08564/) (2020)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [Scheduling Policies for Minimizing Age of Information in Broadcast Wireless Networks](https://mlsanity.com/paper/1801.01803/) (2018)
- [A Whittle Index Approach to Minimizing Functions of Age of Information](https://mlsanity.com/paper/1908.10438/) (2019)
- [Sampling for Data Freshness Optimization: Non-linear Age Functions](https://mlsanity.com/paper/1812.07241/) (2018)
- [Urgency of Information for Context-Aware Timely Status Updates in Remote Control Systems](https://mlsanity.com/paper/2002.07987/) (2020)
- [Timely Communications for Remote Inference](https://mlsanity.com/paper/2404.16281/) (2024)
- [Optimizing Information Freshness in Wireless Networks under General Interference Constraints](https://mlsanity.com/paper/1803.06467/) (2018)
- [FORK: A Forward-Looking Actor For Model-Free Reinforcement Learning](https://mlsanity.com/paper/2010.01652/) (2020)
- [The Age of Correlated Features in Supervised Learning based Forecasting](https://mlsanity.com/paper/2103.00092/) (2021)
- [Monitoring Correlated Sources: AoI-based Scheduling is Nearly Optimal](https://mlsanity.com/paper/2312.16813/) (2023)
- [On the Age of Information in Internet of Things Systems with Correlated Devices](https://mlsanity.com/paper/2001.11162/) (2020)
- [Timely Monitoring of Dynamic Sources with Observations from Multiple Wireless Sensors](https://mlsanity.com/paper/2012.12179/) (2020)
- [How Does Data Freshness Affect Real-time Supervised Learning?](https://mlsanity.com/paper/2208.06948/) (2022)
- [Minimizing the Age of Information from Sensors with Common Observations](https://mlsanity.com/paper/1811.06453/) (2018)
- [Learning and Communications Co-Design for Remote Inference Systems: Feature Length Selection and Transmission Scheduling](https://mlsanity.com/paper/2308.10094/) (2023)
- [On the Monotonicity of Information Aging](https://mlsanity.com/paper/2403.03380/) (2024)
- [Optimizing Age of Information with Correlated Sources](https://mlsanity.com/paper/2208.01738/) (2022)
- [Optimal Status Updates for Minimizing Age of Correlated Information in IoT Networks with Energy Harvesting Sensors](https://mlsanity.com/paper/2310.19216/) (2023)
- [Goal-Oriented Communications for Remote Inference under Two-Way Delay with Memory](https://mlsanity.com/paper/2311.11143/) (2023)
- [Goal-Oriented Status Updating for Real-time Remote Inference over Networks with Two-Way Delay](https://mlsanity.com/paper/2410.08706/) (2024)

## Built on by (most cited first, 1 of 1)

- [QoS-Aware Token Scheduling and Private Data Valuation for Multi-Modal Agentic Networks](https://mlsanity.com/paper/2606.15573/) (2026)

## Abstract (arXiv)

We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model's inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (two or more modalities), we develop an optimal switching-type policy (OPT-ST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies: the round-robin-like switching-type policy (RR-ST) and the round-robin-like threshold policy (RR-Thres). Numerical results show that OPT-ST achieves the lowest inference error among all baselines, with reductions of up to 80.3%. In a five-modality case, RR-ST and RR-Thres achieve 5.2× and 89.8× speedups in offline computation over OPT-ST, respectively, at the cost of increasing the inference error by 23.3% and 38.6%, respectively.