mlsanityData to 14 Oct 2026

Taking Tokenization off the Host for Agentic LLM Serving

arXiv 2610.13325 ↗First posted 9 Oct 2026cs.DC cs.CL cs.LGcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Agentic workloads resend long contexts that barely change between calls. Prefix caching now skips most of their prefill, yet every request must still be tokenized in full before the cache can even be consulted. Tokenization, a CPU stage long treated as negligible, becomes the largest part of time-to-first-token (TTFT), and a frontend that tokenizes for a whole GPU fleet limits how many GPUs it can feed. We present GraphTok, a GPU tokenizer that runs the entire encoding pipeline, from pre-tokenization through byte-pair encoding (BPE) merging to token-ID assembly, as one replayable CUDA graph. GraphTok reproduces the reference token IDs bit-for-bit on GPT-2, Llama 3, Qwen 2.5, DeepSeek-V3, and Gemma 3, hands short inputs to a CPU engine where the GPU cannot win, and plugs into unmodified vLLM and NVIDIA Dynamo. GraphTok encodes one million GPT-2 tokens in 1.7-3.2 ms including data transfers, and on Llama 3 it is 4.7-6.0x faster than gigatoken, the strongest CPU tokenizer. On TokTier's own single-request benchmark, it is 2.6-3.9x faster than gigatoken and up to 18.1x faster than TokTier. In vLLM with a cached prefix, it lowers mean TTFT by 2.1-2.4x. The largest gain is capacity. Within a 2 s P99 target, one Dynamo frontend with GraphTok sustains 518 requests/s of 121k-token traffic, the busiest-hour load of 56 GPUs, against 296 with gigatoken and 18.5 with the default tokenizer. Across 121k to one million tokens, this is 1.75-2.5x the load gigatoken sustains, with a P99 up to 64x lower where gigatoken's tail breaks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 41 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

20 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  2. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan

Other references · 18

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2020Fast WordPiece Tokenizationcited by 87
  6. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  8. 2023A Formal Perspective on Byte-Pair Encodingcited by 36
  9. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16
  10. 2025BlockBPE: Parallel BPE Tokenizationcited by 4
  11. 2026TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Servingcited by 4
  12. 2026GPUTOK: GPU Accelerated Byte Level BPE Tokenizationcited by 3
6 more
  1. 2026Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inferencecited by 3
  2. 2026DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scalecited by 3
  3. 2026Incremental BPE Tokenizationcited by 2
  4. 2026Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Frameworkcited by 2
  5. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNICcited by 1
  6. 2026Dissecting GPU Utilization for LLM Inference on Nvidia Hoppercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
  2. 2026Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
  3. 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
  4. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  5. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
  6. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  7. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
  8. 2026Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving