Nalar: An agent serving framework
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
LLM-driven agentic applications increasingly automate complex, multi-step tasks, but serving them efficiently remains challenging due to heterogeneous components, dynamic and model-driven control flow, long-running state, and unpredictable latencies. Nalar is a ground-up agent-serving framework that cleanly separates workflow specification from execution while providing the runtime visibility and control needed for robust performance. Nalar preserves full Python expressiveness, using lightweight auto-generated stubs that turn agent and tool invocations into futures carrying dependency and context metadata. A managed state layer decouples logical state from physical placement, enabling safe reuse, migration, and consistent retry behavior. A two-level control architecture combines global policy computation with local event-driven enforcement to support adaptive routing, scheduling, and resource management across evolving workflows. Together, these mechanisms allow Nalar to deliver scalable, efficient, and policy-driven serving of heterogeneous agentic applications without burdening developers with orchestration logic. Across three agentic workloads, Nalar cuts tail latency by 34--74%, achieves up to 2.9× speedups, sustains 80 RPS where baselines fail, and scales to 130K futures with sub-500 ms control overhead.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 38 references that are in the map. First posted on arXiv 8 Jan 2026.
Builds on
17 of 38 references.
Closest ancestors · 2
Same direction, newest first
- 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
Other references · 15
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
- 2023OpenAGI: When LLM Meets Domain Expertscited by 197
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
3 more
- 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
- 2023Reducing Energy Bloat in Large Model Trainingcited by 22
- 2023Blox: A Modular Toolkit for Deep Learning Schedulerscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025A Declarative Language for Building And Orchestrating LLM-Powered Agent Workflows
- 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
- 2026GraphBit: A Graph-based Agentic Framework for Non-Linear Agent Orchestration
- 2026AgentSPEX: An Agent SPecification and EXecution Language
- 2026A Policy-Driven Runtime Layer for Agentic LLM Serving
- 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
- 2026VineLM: Trie-Based Fine-Grained Control for Agentic Workflows
- 2026A Lightweight Modular Framework for Constructing Autonomous Agents Driven by Large Language Models: Design, Implementation, and Applications in AgentForge