# Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Models arXiv: https://arxiv.org/abs/2408.15518 ยท HTML: https://mlsanity.com/paper/2408.15518/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 42 of its 44 references that are in the map. - First posted on arXiv 28 Aug 2024. ## Builds on (most cited first, 30 of 42) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020) - [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019) - [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023) - [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019) - [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023) - [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019) - [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019) - [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023) - [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023) - [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023) - [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021) - [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023) - [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023) - [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021) - [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022) - [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023) - [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023) ## Built on by (most cited first, 6 of 6) - [AutoNeural: Co-Designing Vision-Language Models for NPU Inference](https://mlsanity.com/paper/2512.02924/) (2025) - [OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference](https://mlsanity.com/paper/2412.11475/) (2024) - [CAMPHOR: Collaborative Agents for Multi-input Planning and High-Order Reasoning On Device](https://mlsanity.com/paper/2410.09407/) (2024) - [Scaling LLM Pre-training with Vocabulary Curriculum](https://mlsanity.com/paper/2502.17910/) (2025) - [EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment](https://mlsanity.com/paper/2410.05938/) (2024) - [Intelligent IoT Attack Detection Design via ODLLM with Feature Ranking-based Knowledge Base](https://mlsanity.com/paper/2503.21674/) (2025) ## Abstract (arXiv) This paper presents Dolphin, a novel decoder-decoder architecture for energy-efficient processing of long contexts in language models. Our approach addresses the significant energy consumption and latency challenges inherent in on-device models. Dolphin employs a compact 0.5B parameter decoder to distill extensive contextual information into a memory embedding, substantially reducing the input length for the primary 7B parameter decoder model. Inspired by vision-language models, we repurpose the image embedding projector to encode long textual contexts, effectively treating extended context as a distinct modality. This innovative method enables processing of substantially longer contexts without the typical computational overhead associated with extended input sequences. Empirical evaluations demonstrate a 10-fold improvement in energy efficiency and a 5-fold reduction in latency compared to conventional full-length context processing methods without losing quality of the response. Our work contributes to the development of more sustainable and scalable language models for on-device applications, addressing the critical need for energy-efficient and responsive AI technologies in resource-constrained environments while maintaining the accuracy to understand long contexts. This research has implications for the broader field of natural language processing, particularly in the domain of efficient model design for resource-limited settings. By enabling more sophisticated AI capabilities on edge devices, Dolphin paves the way for advanced language processing in a wide range of applications where computational resources are at a premium. The Dolphin model is publicly available at https://huggingface.co/NexaAIDev/Dolphin.