TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
Not on the map. Why?
Abstract (arXiv)
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 69 of its 85 references that are in the map. First posted on arXiv 11 Mar 2026.
Builds on
69 of 85 references.
References · 69
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2023GPT-4 Technical Reportcited by 18,132
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2025Qwen3 Technical Reportcited by 8,195
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2022Emergent Abilities of Large Language Modelscited by 2,483
38 more
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2024Mixtral of Expertscited by 1,785
- 2019Towards VQA Models That Can Readcited by 1,696
- 2025Gemma 3 Technical Reportcited by 1,686
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2025s1: Simple test-time scalingcited by 1,311
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
- 2019Green AIcited by 620
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
- 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
- 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
- 2023A Survey on Model Compression for Large Language Modelscited by 263
- 2022Contrastive Test-Time Adaptationcited by 258
- 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
- 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
- 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
- 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
- 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
- 2020Rotated Binary Neural Networkcited by 77
- 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
- 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
- 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
- 2019DeepShift: Towards Multiplication-Less Neural Networkscited by 52
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
- 2026EinSort: Sorting is All We Need for Tensorizing LLM7 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
- 2023A Speed Odyssey for Deployable Quantization of LLMs
- 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
- 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
- 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
- 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
- 2026Tailoring the Quantization Space for 1-Bit KV Cache Compression
- 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration