mlsanityData to 14 Oct 2026

JustFit: Just-in-Time State Management for Local LLM Serving

arXiv 2609.17475 ↗First posted 15 Sep 2026cs.AI cs.PFcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes. With Qwen3.8-27B MXFP4 on a 24 GiB M4 Pro MacBook, JustFit reaches 327,680 retained positions across two requests, 10.67 times the evaluated baseline's 30,720-position single-request record. One request completes the full 262,144-position native window at median 5.986 tokens/s. Each shape completes 16,384 outputs per request in three fresh processes: B1 uses one cold build and two prefix extensions; B2 uses three ordered prefix extensions (Section 4). Image encoding can proceed while preserving a live 196,608-input text request. A controlled, repetitive 32K+6K workload reaches median 18.284 tokens/s at 15,626 MiB; a separate AIME 2026 evaluation scores 29/30. Coordinating execution and state lifetimes makes longer histories feasible on personal hardware.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 27 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

15 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec

Other references · 13

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  9. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  10. 2026Native LLM and MLLM Inference at Scale on Apple Siliconcited by 5
  11. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4
  12. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 4
1 more
  1. 2026Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Siliconcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  2. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
  3. 2026MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance
  4. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  5. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  6. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  7. 2026Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing
  8. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving