mlsanityData to 7 Oct 2026

Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Models

arXiv 2609.25014 ↗First posted 2 Aug 2026cs.CR cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Organizations fine-tune small language models on private data and then compress them to 4 bits for resource-efficient deployment. We show that the compression method also affects privacy. What separates the methods is not the bit width but whether they tune their rounding on a small sample of text, the calibration corpus. On our primary model, when each planted record's own opening text is used as the prompt, the two calibration-based methods we test, Activation-aware Weight Quantization (AWQ) and Gradient-based Post-Training Quantization (GPTQ), each reproduce none of the planted records, while the calibration-corpus-free GGUF Q4_K_M format reproduces 5.3% of them. Tracked across five open models with 0.5-7 billion parameters, AWQ leaks least at every size and in both families, with little accuracy loss at 3-7 billion. Controlled experiments associate the difference with calibration-induced rounding error in channels involved in rare-token prediction. Choosing the 4-bit method is therefore a deployment-time privacy decision, not only a question of speed and quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 20 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

19 of 20 references.

References · 19

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  7. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  8. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  9. 2024Stealing Part of a Production Language Modelcited by 152
  10. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  11. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  12. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
7 more
  1. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  2. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  3. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  4. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 7
  5. 2025Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRAcited by 3
  6. 2025How Quantization Impacts Privacy Risk on LLMs for Code?cited by 3
  7. 2026Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTscited by 1

Built on by

Not cited yet.