mlsanityData to 14 Oct 2026

On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding

arXiv 2510.15620 ↗First posted 17 Oct 2025cs.LGcited by 1

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Semantic top-K selection with cross-encoder rerankers underpins on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation. However, its latency and memory demands dominate end-to-end budgets on edge hardware. Revisiting the objective of top-K selection, we reveal that only relative rankings matter, not exact per-candidate scores. We further observe sequence-level sparsity: relative rankings progressively stabilize in intermediate layers, enabling early pruning prior to completing full inference. Building on this insight, we propose monolithic forwarding and develop a training-free inference system, PRISM. By maintaining a global view of all candidates, it reduces latency through progressive cluster pruning. It also bounds peak memory usage by strategically overlapping I/O with computation via overlapped layer streaming and chunked execution. We evaluate PRISM against state-of-the-art baselines on rerankers from 0.6 B to 8 B parameters across Apple M2 and RTX 5070. PRISM consistently reduces latency by up to 89.2% and peak memory by up to 91.3% in microbenchmarks, without compromising precision. Across three real-world on-device AI applications, PRISM lowers latency by 11.6%-51.0% and peak memory by 18.6%-77.8%, demonstrating substantial improvements in efficiency and deployability.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 59 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

32 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Hybrid-RACA: Hybrid Retrieval-Augmented Composition Assistance for Real-time Text Prediction8 Aug
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  10. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
18 more
  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  2. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  4. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  5. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  6. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  7. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
  8. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  9. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  10. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  11. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 38
  12. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  13. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAGcited by 16
  14. 2024MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generationcited by 13
  15. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  16. 2025MobiAgent: A Systematic Framework for Customizable Mobile Agentscited by 9
  17. 2025LEANN: A Low-Storage Vector Indexcited by 4
  18. 2024EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Updatecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inference
  2. 2026FLASH-MAXSIM: IO-Aware Fused Kernels for Late-Interaction Retrieval
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  4. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
  5. 2026From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG
  6. 2026MobileMoE: Scaling On-Device Mixture of Experts
  7. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
  8. 2026When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving