# FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence

arXiv: https://arxiv.org/abs/2402.03694 · HTML: https://mlsanity.com/paper/2402.03694/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 78 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [Deep Packet: A Novel Approach For Encrypted Traffic Classification Using Deep Learning](https://mlsanity.com/paper/1709.02656/) (2017)
- [NetDiffusion: Network Data Augmentation Through Protocol-Constrained Traffic Generation](https://mlsanity.com/paper/2310.08543/) (2023)
- [New Directions in Automated Traffic Analysis](https://mlsanity.com/paper/2008.02695/) (2020)
- [Automated Website Fingerprinting through Deep Learning](https://mlsanity.com/paper/1708.06376/) (2017)
- [netFound: Principled Design for Network Foundation Models](https://mlsanity.com/paper/2310.17025/) (2023)
- [Neural Packet Classification](https://mlsanity.com/paper/1902.10319/) (2019)
- [Taurus: A Data Plane Architecture for Per-Packet ML](https://mlsanity.com/paper/2002.08987/) (2020)
- [pForest: In-Network Inference with Random Forests](https://mlsanity.com/paper/1909.05680/) (2019)
- [Measuring the Performance and Network Utilization of Popular Video Conferencing Applications](https://mlsanity.com/paper/2105.13478/) (2021)
- [LEAF: Navigating Concept Drift in Cellular Networks](https://mlsanity.com/paper/2109.03011/) (2021)
- [Large-scale Mobile App Identification Using Deep Learning](https://mlsanity.com/paper/1910.02350/) (2019)
- [Dynamic Domain Adaptation for Efficient Inference](https://mlsanity.com/paper/2103.16403/) (2021)
- [A Lightweight, Efficient and Explainable-by-Design Convolutional Neural Network for Internet Traffic Classification](https://mlsanity.com/paper/2202.05535/) (2022)
- [Active Learning for Network Traffic Classification: A Technical Study](https://mlsanity.com/paper/2106.06933/) (2021)
- [AC-DC: Adaptive Ensemble Classification for Network Traffic Identification](https://mlsanity.com/paper/2302.11718/) (2023)
- [Homunculus: Auto-Generating Efficient Data-Plane ML Pipelines for Datacenter Networks](https://mlsanity.com/paper/2206.05592/) (2022)
- [Estimating WebRTC Video QoE Metrics Without Using Application Headers](https://mlsanity.com/paper/2306.01194/) (2023)
- [Traffic Analytics Development Kits (TADK): Enable Real-Time AI Inference in Networking Apps](https://mlsanity.com/paper/2208.07558/) (2022)

## Built on by (most cited first, 1 of 1)

- [Non-uniformity is All You Need: Efficient and Timely Encrypted Traffic Classification With ECHO](https://mlsanity.com/paper/2406.01852/) (2024)

## Abstract (arXiv)

Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based classification unreliable. At high flow-arrival rates, however, new flows can arrive faster than end-to-end analysis pipelines can process. Unlike typical cloud inference workloads, packets from the same flow arrive over time and require per-flow state. This streaming, stateful setting limits naive scale-out with stateless workers. This paper presents FlowWise, a serving system for streaming, stateful flow intelligence on commodity CPU/GPU servers that jointly selects (i) how many packets to collect and (ii) which model(s) to apply for each flow to balance latency, service rate, and accuracy. Across three representative tasks, inference time across candidate models differs by 1.8x-141.3x, while packet waiting time can exceed feature computation and inference by 6-8 orders of magnitude at the tail. FlowWise exploits these gaps with an uncertainty-driven fast-slow cascade: it issues early predictions from a one-packet model and escalates only uncertain flows to slower models that wait for additional packets and use richer features. A queue-centric runtime maintains per-flow state, performs asynchronous feature joining, and uses brokered, flow-ID-keyed streams to distribute workload across heterogeneous workers. FlowWise serves 76.3% of flows within 16 ms, improving median end-to-end serving latency by 40.5x while increasing service rate and maintaining similar accuracy to always running the slow model. With thousands of packet-header features per flow, our scale-up benchmark sustains over 48.5k new flows per second on a single 16-core commodity CPU server, comparable to reported flow-arrival rates in city-backbone traces. FlowWise is open sourced at https://github.com/noise-lab/FlowWise.