mlsanityData to 7 Oct 2026

Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation

arXiv 2606.01725 ↗First posted 1 Jun 2026cs.AI cs.LGcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agentic AI completes tasks through iterative planning, tool use, and reasoning based on observed outcomes. Despite its popularity, its system-level behavior remains poorly understood, particularly for complex datasets and agent architectures-owing to highly non-deterministic execution, prohibitive evaluation costs, and limited visibility into proprietary models. This paper presents GAIATrace, the first token-level trace dataset of two state-of-the-art agentic systems (MiroThinker and OWL) running GAIA, a benchmark composed of a heterogeneous mix of general-purpose tasks. Unlike prior trace datasets, GAIATrace captures full reasoning tokens, task-level structures, and activities of every major participating LLMs, enabling in-depth systems research. Complementing the dataset, we present Vidur-Agent, a trace-driven simulator that can replay GAIATrace to perform reproducible, low-cost system evaluation across diverse simulated environments. Using both artifacts, we characterize how modern agentic systems handle general tasks and how various system design choices shape their behavior, yielding several unique findings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 73 of its 99 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

73 of 99 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective4 Jun
  2. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation29 May
  3. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios22 May
  4. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems1 Apr
  5. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
  6. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Execution5 Nov
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
38 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  11. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  12. 2023An LLM Compiler for Parallel Function Callingcited by 170
  13. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  14. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  15. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  16. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  17. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  18. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  19. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  20. 2023Fairness in Serving Large Language Modelscited by 77
  21. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 72
  22. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
  23. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  24. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  25. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  26. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  27. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  28. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  29. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  30. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  31. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  32. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  33. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  34. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scalecited by 13
  35. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  36. 2026MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Taskscited by 13
  37. 2025Efficient and Scalable Agentic AI with Heterogeneous Systemscited by 10
  38. 2025FlowSearch: Advancing deep research with dynamic structured knowledge flowcited by 10

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agentscited by 0
  2. 2026PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstructioncited by 0
  3. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
  2. 2026TRACE: Capability-Targeted Agentic Training
  3. 2025Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems
  4. 2026TriCEGAR: A Trace-Driven Abstraction Mechanism for Agentic AI
  5. 2026From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation
  6. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  7. 2025ARE: Scaling Up Agent Environments and Evaluations
  8. 2026Agentic AI Workload Characteristics