# Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI

arXiv: https://arxiv.org/abs/2507.11830 · HTML: https://mlsanity.com/paper/2507.11830/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models](https://mlsanity.com/paper/2405.05374/) (2024)
- [SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation](https://mlsanity.com/paper/2410.03960/) (2024)

## Built on by (most cited first, 2 of 2)

- [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](https://mlsanity.com/paper/2602.22593/) (2026)
- [Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads](https://mlsanity.com/paper/2509.16495/) (2025)

## Abstract (arXiv)

Inference is now the dominant AI workload, yet existing systems force trade-offs between latency, throughput, and cost. Arctic Inference, an open-source vLLM plugin from Snowflake AI Research, introduces Shift Parallelism, a dynamic parallelism strategy that adapts to real-world traffic while integrating speculative decoding, SwiftKV compute reduction, and optimized embedding inference. It achieves up to 3.4 times faster request completion, 1.75 times faster generation, and 1.6M tokens/sec per GPU for embeddings, outperforming both latency- and throughput-optimized deployments. Already powering Snowflake Cortex AI, Arctic Inference delivers state-of-the-art, cost-effective inference for enterprise AI and is now available to the community.