mlsanityData to 14 Oct 2026

WindVE: Collaborative CPU-NPU Vector Embedding

arXiv 2504.14941 ↗First posted 21 Apr 2025cs.DCcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highly sensitive to cost-performance ratio, prompting the need for improving hardware resource utilization in the inference service. Specifically, vector embedding and retrieval processes take up to 20% of the total latency. Therefore, optimizing the utilization of computational resources in vector embeddings is crucial for enhancing the cost-performance ratio of inference processes, which in turn boosts their product competitiveness.In this paper, we analyze the deployment costs of vector embedding technology in inference services, propose a theoretical formula, and determine through the mathematical expression that increasing the capacity to process concurrent queries is the key to reducing the deployment costs of vector embeddings. Therefore, in this paper, we focus on improving the product's capability to process concurrent queries. To optimize concurrency without sacrificing performance, we have designed a queue manager that adeptly offloads CPU peak queries. This manager utilizes a linear regression model to ascertain the optimal queue depths, a critical parameter that significantly influences the efficacy of the system. We further develop a system named WindVE that uses a CPU-NPU heterogeneous architecture to offload peak concurrent queries, which leverages the performance differences between the two processors to effectively manage traffic surges. Through experiments, we compare WindVE to the state-of-the-art vector embedding framework FlagEmbedding, and achieve a concurrency level up to 22.3% higher than the scheme without offloading.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 51 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

21 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Bridging the Preference Gap between Retrievers and LLMs13 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 19

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  5. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  6. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 104
  7. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  8. 2019Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systemscited by 68
  9. 2019GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embeddingcited by 40
  10. 2020Energy-Efficient Resource Management for Federated Edge Learning with CPU-GPU Heterogeneous Computingcited by 39
  11. 2019Fast and Accurate Network Embeddings via Very Sparse Random Projectioncited by 29
  12. 2024Context Embeddings for Efficient Answer Generation in RAGcited by 26
7 more
  1. 2023IAG: Induction-Augmented Generation Framework for Answering Reasoning Questionscited by 15
  2. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  3. 2021Binary Code based Hash Embedding for Web-scale Applicationscited by 11
  4. 2024VectorSearch: Enhancing Document Retrieval with Semantic Embeddings and Optimized Searchcited by 10
  5. 2024HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devicescited by 9
  6. 2022Reads2Vec: Efficient Embedding of Raw High-Throughput Sequencing Reads Datacited by 7
  7. 2023CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Modelscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
  2. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
  3. 2026Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite
  4. 2024Accelerating Retrieval-Augmented Generation
  5. 2026Efficient Iterative Retrieval with Heterogeneous Batching
  6. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  7. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  8. 2026D-NOVA: In-Storage Retrieval Accelerator via Dual-Bound 3D NAND-Optimized Similarity Search with Vector Adaptation