mlsanityData to 14 Oct 2026

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

arXiv 2511.06516 ↗First posted 9 Nov 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Many LLM applications require only narrow capabilities, yet standard post-training quantization (PTQ) methods allocate precision without considering the target task. This can waste bits on layers that are less relevant to the task signal while over-compressing layers that are critical for downstream behavior. We propose Task-Aware Quantization (TAQ), a training-free, weight-only mixed-precision PTQ framework that uses a small set of unlabeled task calibration prompts to allocate higher precision to task-relevant transformer layers under a fixed bit budget. TAQ estimates layer importance from hidden representations and output sensitivity, and we instantiate it with three scoring rules: TAQ-IS, based on activation information and stability; TAQ-KL, based on output-distribution sensitivity under a quantization-noise proxy; and TAQ-O, a label-informed oracle diagnostic for analyzing layer sensitivity. Across several benchmarks, TAQ outperforms task-agnostic baselines such in most settings, with especially strong gains in the accuracy--memory ratio. We further validate that these gains translate to real deployment behavior through hardware throughput and latency measurements, and analyze calibration robustness and residual-stream error propagation. Overall, TAQ turns mixed-precision PTQ from a model-centric compression step into a task-conditioned precision-allocation problem. A reference implementation is available at hrefhttps://anonymous.4open.science/r/TAQ-9217/README.mdincludegraphics[height=1em]imgs/github-mark.png.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 70 of its 91 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

70 of 91 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  15. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  16. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  17. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  18. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  19. 2022Editing Models with Task Arithmeticcited by 1,120
  20. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  21. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  22. 2023Steering Language Models With Activation Engineeringcited by 922
  23. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  24. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  25. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  26. 2024Phi-4 Technical Reportcited by 711
  27. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  28. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  29. 2022Efficiently Scaling Transformer Inferencecited by 539
  30. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  31. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  32. 2021A White Paper on Neural Network Quantizationcited by 472
  33. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  34. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  35. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  36. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  37. 2023In-Context Learning Creates Task Vectorscited by 312
  38. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Activation Steering for Masked Diffusion Language Modelscited by 9
  2. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  3. 2026The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globallycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
  2. 2026Enhancing Post-Training Quantization via Future Activation Awareness
  3. 2025RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
  5. 2021Post-Training Sparsity-Aware Quantization
  6. 2023Benchmarking the Reliability of Post-training Quantization: a Particular Focus on Worst-case Performance
  7. 2026The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization
  8. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training