AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications
Not on the map. Why?
Abstract (arXiv)
Large language model (LLM) applications increasingly operate as streaming workflows combining retrieval, tool calls, safety filters, and multi-agent coordination. Although contemporary frameworks expose provider deltas, workflow nodes often treat generation as coarse request-response steps, leaving queue management, worker allocation, ordering, and backpressure to ad hoc callback code. This paper presents AiFlow, a token-native reactive orchestration model that normalizes provider deltas into typed Context<T> events propagated through a directed streaming graph. Each node is managed by a Node Guardian that declares and enforces local queue bounds, worker concurrency, ordering, overflow policy, cancellation propagation, and retry discipline. We formalize the bounded-memory property, present the compilation from a compact DSL and JSON graph form, and provide static validation for type safety, state concurrency, and injection compatibility. Controlled microbenchmarks, captured DeepSeek trace replay (30 runs), descriptive online runs, LangGraph baselines, a streaming RAG workload, and an Ollama local-backend check show that AiFlow does not alter provider-side Model TTFT but reduces Application TTFPT by 70.9-94.7% versus aggregation and keeps runtime-owned queue depth within declared bounds (93.7-96.5% MaxQ reduction versus unbounded policies). The supplementary artifact contains scripts, raw traces, machine-readable tables, checksums, and an API-free smoke test; the public implementation is available through the FIT Framework repository.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 40 references that are in the map. First posted on arXiv 1 Aug 2026.
Builds on
12 of 40 references.
References · 12
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
- 2024An Empirical Study on Challenges for LLM Application Developerscited by 17
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Substrate-Portable Execution for Production LLM Workflows
- 2026HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
- 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
- 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
- 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
- 2025Accelerating Language Model Workflows with Prompt Choreography
- 2026Separating Intelligence from Execution: A Workflow Engine for the Model Context Protocol
- 2026Stage-Aware Communication Scheduling for Disaggregated LLM Serving