mlsanityData to 14 Oct 2026

LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs

arXiv 2505.24451 ↗First posted 30 May 2025cs.CR cs.AIcited by 4

On the map: Mechanistic interpretability → Steering and probing for LLM safety

Abstract (arXiv)

Large Language Models (LLMs) are being extensively used for cybersecurity purposes. One of them is the detection of vulnerable codes. For the sake of efficiency and effectiveness, compression and fine-tuning techniques are being developed, respectively. However, they involve spending substantial computational efforts. In this vein, we analyse how Linear Probes (LPs) can be used to provide an estimation on the performance of a compressed LLM at an early phase -- before fine-tuning. We also show their suitability to set the cut-off point when applying layer pruning compression. Our approach, dubbed LPASS, is applied in BERT and Gemma for the detection of 12 of MITRE's Top 25 most dangerous vulnerabilities on 480k C/C++ samples. LPs can be computed in 142.97 s. and provide key findings: (1) 33.3 % and 72.2% of layers can be removed, respectively, with no precision loss; (2) they provide an early estimate of the post-fine-tuning and post-compression model effectiveness, with 3% and 8.68% as the lowest and average precision errors, respectively. LPASS-based LLMs outperform the state of the art, reaching 86.9% of accuracy in multi-class vulnerability detection. Interestingly, LPASS-based compressed versions of Gemma outperform the original ones by 1.6% of F1-score at a maximum while saving 29.4 % and 23.8% of training and inference time and 42.98% of model size.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Steering and probing for LLM safety, one of 13 directions in Mechanistic interpretability. Builds on 35 of its 58 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

35 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Linear Probe Penalties Reduce LLM Sycophancy1 Dec

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  10. 2023A Survey on Model Compression for Large Language Modelscited by 269
  11. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  12. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
22 more
  1. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  2. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 102
  3. 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 52
  4. 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
  5. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 47
  6. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 46
  7. 2024Large Language Model for Vulnerability Detection: Emerging Results and Future Directionscited by 45
  8. 2023ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?cited by 44
  9. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  10. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 40
  11. 2021What do pre-trained code models know about code?cited by 36
  12. 2022LineVD: Statement-level Vulnerability Detection using Graph Neural Networkscited by 35
  13. 2023How Far Have We Gone in Vulnerability Detection Using Large Language Modelscited by 27
  14. 2022Compressing Pre-trained Models of Code into 3 MBcited by 21
  15. 2023Greening Large Language Models of Codecited by 19
  16. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  17. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  18. 2023Energy and Carbon Considerations of Fine-Tuning BERTcited by 11
  19. 2024Non-Linear Inference Time Intervention: Improving LLM Truthfulnesscited by 9
  20. 2023The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Modelscited by 8
  21. 2023FLAG: Finding Line Anomalies (in code) with Generative AIcited by 7
  22. 2023Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Modelscited by 8
  2. 2026The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicatorscited by 2
  3. 2026Activation Probes Surface Code-Security Signals that the Model's Output Missescited by 0
  4. 2026Probing the Prefill: Detecting Code Vulnerabilities via Latent Activationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Local Sparsity Enables Unsupervised LLM Safety Detection
  2. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
  3. 2023Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation
  4. 2025LLMpatronous: Harnessing the Power of LLMs For Vulnerability Detection
  5. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
  6. 2025"Digital Camouflage": The LLVM Challenge in LLM-Based Malware Detection
  7. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security
  8. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis