# Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference

arXiv: https://arxiv.org/abs/2508.13380 · HTML: https://mlsanity.com/paper/2508.13380/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Task Adaptive Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2203.16708/) (2022)
- [Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints](https://mlsanity.com/paper/1311.2106/) (2013)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution](https://mlsanity.com/paper/1812.10027/) (2018)
- [Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection](https://mlsanity.com/paper/2006.07864/) (2020)
- [QoS-Aware Placement of Deep Learning Services on the Edge with Multiple Service Implementations](https://mlsanity.com/paper/2104.15094/) (2021)
- [A Multi-task Supervised Compression Model for Split Computing](https://mlsanity.com/paper/2501.01420/) (2025)
- [Joint Optimization of Service Caching Placement and Computation Offloading in Mobile Edge Computing Systems](https://mlsanity.com/paper/1906.00711/) (2019)
- [AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learning](https://mlsanity.com/paper/2304.08594/) (2023)
- [Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterization](https://mlsanity.com/paper/1912.06322/) (2019)
- [Joint Batching and Scheduling for High-Throughput Multiuser Edge AI with Asynchronous Task Arrivals](https://mlsanity.com/paper/2307.14350/) (2023)

## Built on by (most cited first, 1 of 1)

- [Online Learning for Multi-Layer Hierarchical Inference under Partial and Policy-Dependent Feedback](https://mlsanity.com/paper/2603.04247/) (2026)

## Abstract (arXiv)

The growing demand for intelligent services on resource-constrained edge devices has spurred the development of collaborative inference systems that distribute workloads across end devices, edge servers, and the cloud. While most existing frameworks focus on single-task, single-model scenarios, many real-world applications (e.g., autonomous driving and augmented reality) require concurrent execution of diverse tasks including detection, segmentation, and depth estimation. In this work, we propose a unified framework to jointly decide which multi-task models to deploy (onload) at clients and edge servers, and how to route queries across the hierarchy (offload) to maximize overall inference accuracy under memory, compute, and communication constraints. We formulate this as a mixed-integer program and introduce J3O (Joint Optimization of Onloading and Offloading), an alternating algorithm that (i) greedily selects models to onload via Lagrangian-relaxed submodular optimization and (ii) determines optimal offloading via constrained linear programming. We further extend J3O to account for batching at the edge, maintaining scalability under heterogeneous task loads. Experiments show J3O consistently achieves over 97% of the optimal accuracy while incurring less than 15% of the runtime required by the optimal solver across multi-task benchmarks.