mlsanityData to 7 Oct 2026

Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference

arXiv 2406.07007 ↗First posted 11 Jun 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. On-device LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple question-answer datasets, and show the efficacy of our method in the LLM customization.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

21 of 34 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
9 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  4. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  5. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  6. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  7. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  8. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  9. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 17
  2. 2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 13
  3. 2025Chain-of-Rank: Enhancing Large Language Models for Domain-Specific RAG in Edge Devicecited by 2
  4. 2025Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performancecited by 2
  5. 2025Compressing Many-Shots in In-Context Learningcited by 1
  6. 2025Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAGcited by 1
  7. 2025MobileFineTuner: A Mobile-Native Framework for On-Device LLM Fine-Tuning in Real-World Embedded AI Applicationscited by 1