# Lightweight Chunk Selection for Mobile Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2608.03148 · HTML: https://mlsanity.com/paper/2608.03148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [End-to-End Neural Ad-hoc Ranking with Kernel Pooling](https://mlsanity.com/paper/1706.06613/) (2017)
- [Learning to Filter Context for Retrieval-Augmented Generation](https://mlsanity.com/paper/2311.08377/) (2023)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [Multilingual Routing in Mixture-of-Experts](https://mlsanity.com/paper/2510.04694/) (2025)
- [Provence: efficient and robust context pruning for retrieval-augmented generation](https://mlsanity.com/paper/2501.16214/) (2025)
- [NeuroMoE: A Transformer-Based Mixture-of-Experts Framework for Multi-Modal Neurological Disorder Classification](https://mlsanity.com/paper/2506.14970/) (2025)
- [Pocket RAG: On-Device RAG for First Aid Guidance in Offline Mobile Environment](https://mlsanity.com/paper/2602.13229/) (2026)
- [One Model Is Enough: Native Retrieval Embeddings from LLM Agent Hidden States](https://mlsanity.com/paper/2603.08429/) (2026)

## Abstract (arXiv)

RAG improves the factual grounding of LLM by incorporating external knowledge, but deploying RAG on mobile and edge devices remains challenging because retrieved context increases computation and memory. A direct way to reduce this cost is to retain only one retrieved chunk before generation, but the top-ranked retrieved chunk is not always the most evidence-supporting one, since retrieval similarity does not necessarily imply evidential sufficiency. Existing context-reduction methods can improve context quality, but often require additional LLMs or compressors that are costly under a strict mobile budget. In this paper, we study lightweight RAG chunk selection as an evidence-alignment problem. Our selector combines three complementary feature sources: question hidden states that represent LLM-side query intent, MoE routing-derived expert signals that capture the generator's internal routing structure, and retrieved chunk embeddings that preserve candidate-side evidence geometry. A compact multilayer perceptron maps these features to an evidence prototype in the chunk embedding space, and the candidate most aligned with this prototype is selected by cosine similarity. For stricter deployment budgets, we further introduce an optional task-aware feature selection strategy to reduce the selector input dimension. To support supervised evaluation, we construct semantic chunk-correctness labels based on evidence sufficiency rather than answer-string containment. Experiments show that the proposed selector consistently improves rank-1 evidence selection over mobile-applicable baselines by an average of 2.5%. These results suggest that using LLM-side query representations and MoE routing information and aligning them with retrieval-side candidate embedding is an effective and parameter-efficient strategy for mobile-applicable RAG chunk selection.