# Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs

arXiv: https://arxiv.org/abs/2509.00096 · HTML: https://mlsanity.com/paper/2509.00096/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space](https://mlsanity.com/paper/2402.17811/) (2024)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Truth is Universal: Robust Detection of Lies in LLMs](https://mlsanity.com/paper/2407.12831/) (2024)
- [Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://mlsanity.com/paper/2403.15447/) (2024)
- [SlimGPT: Layer-wise Structured Pruning for Large Language Models](https://mlsanity.com/paper/2412.18110/) (2024)
- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- [Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.18248/) (2024)
- [Small Language Model Can Self-correct](https://mlsanity.com/paper/2401.07301/) (2024)
- [Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models](https://mlsanity.com/paper/2403.02178/) (2024)
- [Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning](https://mlsanity.com/paper/2410.07461/) (2024)
- [Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment](https://mlsanity.com/paper/2503.18991/) (2025)
- [FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression](https://mlsanity.com/paper/2507.20030/) (2025)
- [When Truthful Representations Flip Under Deceptive Instructions?](https://mlsanity.com/paper/2507.22149/) (2025)
- [Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs](https://mlsanity.com/paper/2508.19432/) (2025)

## Built on by (most cited first, 2 of 2)

- [Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2](https://mlsanity.com/paper/2512.22671/) (2025)
- [VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search](https://mlsanity.com/paper/2510.15948/) (2025)

## Abstract (arXiv)

Neural network pruning has emerged as a promising approach for deploying LLMs in low-resource scenarios while preserving downstream task performance. However, for the first time, we reveal that such pruning disrupts LLMs' internal activation features crucial for lie detection, where probing classifiers (typically small logistic regression models) trained on these features assess the truthfulness of LLM-generated statements. This discovery raises a crucial open question: how can we prune LLMs without sacrificing these critical lie detection capabilities? Our investigation further reveals that naively adjusting layer-wise pruning sparsity based on importance inadvertently removes crucial weights, failing to improve lie detection performance despite its reliance on the most crucial LLM layer. To address this issue, we propose Truthful Pruning aligned by Layer-wise Outliers (TPLO), which places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously. This preserves LLMs' original performance while retaining critical features of inner states needed for robust lie detection. Moreover, we introduce a prompting rule to enrich the TruthfulQA benchmark for better calibrating LLM pruning. Empirical results show that our approach improves the hallucination detection for pruned LLMs (achieving 88% accuracy at 50% sparsity) and enhances their performance on TruthfulQA.