mlsanityData to 14 Oct 2026

TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination

arXiv 2510.22767 ↗First posted 26 Oct 2025cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) typically come with a fixed architecture, despite growing evidence that not all layers contribute equally to every downstream task. We introduce TALE (Task-Aware Layer Elimination), an inference-time method that improves task performance by selectively removing layers that are irrelevant or detrimental for a given task. TALE optimizes task-specific performance, yielding a task-optimized architecture without retraining. Across 9 tasks and 5 model families, under both zero-shot and few-shot settings, TALE consistently matches or surpasses baseline performance while simultaneously reducing computational costs. TALE also synergizes with fine-tuning, leading to further performance improvements. Computing TALE for a new task requires modest resources, making it a practical and deployable solution for task-specialized LLM inference.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

45 of 65 references.

References · 45

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2016Deep Variational Information Bottleneckcited by 930
33 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  8. 2018MINE: Mutual Information Neural Estimationcited by 683
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  13. 2021Block Pruning For Faster Transformerscited by 177
  14. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  15. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  16. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  17. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  18. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  19. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  20. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  21. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  22. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  23. 2018Entropy and mutual information in models of deep neural networkscited by 51
  24. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  25. 2013Equitability, mutual information, and the maximal information coefficientcited by 45
  26. 2016Compression of Neural Machine Translation Models via Pruningcited by 44
  27. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  28. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  29. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
  30. 2023E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsitycited by 15
  31. 2025The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generationcited by 10
  32. 2025Analyzing limits for in-context learningcited by 6
  33. 2024Re-examining learning linear functions in contextcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TAPIOCA: Why Task- Aware Pruning Improves OOD model Capabilitycited by 0
  2. 2026What Does Layer-Importance Reveal About Transformers and State-Space Models?cited by 0