mlsanityData to 7 Oct 2026

Task-Centric Acceleration of Small-Language Models

arXiv 2602.24174 ↗First posted 27 Feb 2026cs.CL cs.AI cs.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Small language models (SLMs) have emerged as efficient alternatives to large language models for task-specific applications. However, they are often employed in high-volume, low-latency settings, where efficiency is crucial. We propose TASC, Task-Adaptive Sequence Compression, a framework for SLM acceleration comprising two use-cases: When performing SLM fine-tuning, we propose TASC-ft, which iteratively enriches the tokenizer vocabulary with high-frequency output n-grams and then fine-tunes the model to utilize the expanded vocabulary. Next, we propose an inference-time method, termed TASC-spec. TASC-spec is a lightweight, training-free speculative decoding method that constructs an n-gram draft model from the task's output corpus, mixing task and context n-gram information.TASC-spec avoids any additional training, while bypassing draft-target vocabulary alignment constraints. We demonstrate the effectiveness of both methods across multiple low output-variability generation tasks. Our methods show consistent improvements in inference efficiency while maintaining task performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 67 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

37 of 67 references.

References · 37

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
25 more
  1. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  3. 2025Small Language Models are the Future of Agentic AIcited by 254
  4. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  6. 2019A Question-Entailment Approach to Question Answeringcited by 133
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2019Large-Scale Multi-Label Text Classification on EU Legislationcited by 104
  9. 2023Tokenization and the Noiseless Channelcited by 58
  10. 2025SuperBPE: Space Travel for Language Modelscited by 58
  11. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  12. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  13. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  14. 2023An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Textcited by 23
  15. 2024Understanding Transformers via N-gram Statisticscited by 23
  16. 2021AVocaDo: Strategy for Adapting Vocabulary to Downstream Domaincited by 20
  17. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabulariescited by 20
  18. 2024Multi-word Tokenization for Sequence Compressioncited by 16
  19. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  20. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  21. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 13
  22. 2025Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performancecited by 12
  23. 2025Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offscited by 11
  24. 2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
  25. 2022Data Augmentation for Biomedical Factoid Question Answeringcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
  2. 2026SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
  3. 2025Automatic Task Detection and Heterogeneous LLM Speculative Decoding
  4. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  5. 2024Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs
  6. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  7. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  8. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution