mlsanityData to 14 Oct 2026

Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference

arXiv 2606.31093 ↗First posted 30 Jun 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multimodal models increasingly integrate heterogeneous components, from encoders and LLMs to diffusion models and media decoders. Serving these models efficiently requires flexible workflow orchestration, independent component scheduling, and cross-component state sharing. However, existing multimodal frameworks primarily organize execution as stage-level pipelines, while state ownership remains largely stage-local. We present Omni-Flow, a distributed serving framework built around three cooperating abstractions. Control Flow defines workflows through a Python DSL, organizing heterogeneous components and their dependencies into a unified dataflow graph. The runtime uses these dependencies to determine when each component can execute. Data Flow manages the placement, transfer, and lifetime of shared tensors and paged KV caches across roles, together with same-device weight sharing. Compute Flow matches multimodal conversation histories to reuse KV across turns and integrates framework-managed KV and model-specific sampling with SGLang's execution interfaces. Omni-Flow also enables cross-role prefix-KV reuse between compatible LLM and diffusion components. Together, these abstractions separate model-specific computation from workflow coordination and state management, allowing independently executing roles to share compatible resources through a common programming model. Experiments with Qwen3-Omni show that, across four benchmarks and multiple concurrency levels, Omni-Flow's aggregate job completion time (JCT) is 1.2% and 4.9% higher than vLLM-Omni's and SGLang-Omni's, respectively. In a 50-turn multimodal session, Omni-Flow avoids resubmitting accumulated inputs through server-side session state, reducing mean JCT by 7.5% and 19.3% relative to vLLM-Omni and SGLang-Omni, respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 72 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

43 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2020Longformer: The Long-Document Transformercited by 3,413
25 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  9. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  13. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  14. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  15. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  16. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  17. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  18. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  19. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  20. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  21. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  22. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  23. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  24. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  25. 2026vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Modelscited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism
  2. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
  3. 2025FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference
  4. 2026M*: A Modular, Extensible, Serving System for Multimodal Models
  5. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
  6. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
  7. 2026UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Models
  8. 2026EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend