mlsanityData to 7 Oct 2026

Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

arXiv 2603.06403 ↗First posted 6 Mar 2026cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language model (MLLM) inference scheduling enables strong response quality under practical and heterogeneous budgets, beyond what a homogeneous single-backend setting can offer. Yet online MLLM task scheduling is nontrivial, as requests vary sharply in modality composition and latent reasoning difficulty, while execution backends incur distinct, time-varying costs due to system jitter and network variation. These coupled uncertainties pose two core challenges: deriving semantically faithful yet scheduling-relevant multi-modal task representations, and making low-overhead online decisions over irreversible multi-dimensional budgets. Accordingly, we propose M-CMAB (underlineMulti-modal underlineMulti-constraint underlineContextual underlineMulti-underlineArmed underlineBandit), a multi-adapter-enhanced MLLM inference scheduling framework with three components: (i) a CLS-attentive, frozen-backbone Predictor that extracts compact task representations and updates only lightweight adapters for action-specific estimation; (ii) a primal-dual Constrainer that maintains online Lagrange multipliers to enforce long-horizon constraints via per-round objectives; and (iii) a two-phase Scheduler that balances exploration and exploitation under irreversible budgets. We establish a regret guarantee under multi-dimensional knapsack constraints. On a composite multimodal benchmark with heterogeneous backends, M-CMAB consistently outperforms state-of-the-art baselines across budget regimes, achieving up to 14.18% higher reward and closely tracking an oracle-aided upper bound. Codes are available at https://anonymous.4open.science/r/M2CMAB/.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

27 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025On Path to Multimodal Generalist: General-Level and General-Bench7 May
  2. 2025Learning to Inference Adaptively for Multimodal Large Language Models13 Mar
  3. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach1 Feb
  4. 2024Ranked from Within: Ranking Large Multimodal Models Without Labels9 Dec
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2016A Diagram Is Worth A Dozen Imagescited by 793
  3. 2019Introduction to Online Convex Optimizationcited by 793
  4. 2018CoQA: A Conversational Question Answering Challengecited by 674
  5. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  6. 2015Linear Contextual Bandits with Knapsackscited by 82
  7. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routingcited by 32
  8. 2015Algorithms with Logarithmic or Sublinear Regret for Constrained Contextual Banditscited by 29
  9. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  10. 2024Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Banditscited by 26
  11. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23
  12. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
10 more
  1. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challengescited by 14
  2. 2025Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversationscited by 13
  3. 2022Optimal Contextual Bandits with Knapsacks under Realizability via Regression Oraclescited by 12
  4. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
  5. 2025GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Predictioncited by 7
  6. 2024Towards the Causal Complete Cause of Multi-Modal Representation Learningcited by 5
  7. 2025Offline Learning for Combinatorial Multi-armed Banditscited by 5
  8. 2025MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inferencecited by 4
  9. 2024PPVF: An Efficient Privacy-Preserving Online Video Fetching Framework with Correlated Differential Privacycited by 1
  10. 2024BGTplanner: Maximizing Training Accuracy for Differentially Private Federated Recommenders via Strategic Privacy Budget Allocationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  2. 2026POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
  2. 2026TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference
  3. 2026End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
  4. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Models
  5. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
  6. 2025Concise Reasoning in the Lens of Lagrangian Optimization
  7. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  8. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models