mlsanityData to 14 Oct 2026

Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering

arXiv 2510.17015 ↗First posted 19 Oct 2025cs.LG cs.AI cs.DCcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the scheduler is expected to attain fast agent completion with guaranteed worst-case performance. For that objective, our insight is to selectively pampering agents based on their completion order under idealized fair-sharing. We design Justitia, a fair and also efficient scheduler for task-parallel LLM agents. Noticing that memory is prevalently a bottleneck in LLM serving, Justitia quantifies the true agent cost in a memory-centric manner. It also adopts a light-weight yet accurate method to predict agent costs. Finally, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse agents show that it can substantially enhance the scheduling efficiency with fairness preserved.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 48 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

15 of 48 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  7. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  8. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  10. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  11. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  12. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
3 more
  1. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  2. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  3. 2024Ensuring Fair LLM Serving Amid Diverse Applicationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  2. 2026Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
  2. 2026LOCAL: Enabling Learning On-device Contiguously for Agent LLMs
  3. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  5. 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
  6. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
  7. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
  8. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Serving