DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models
Not on the map. Why?
Abstract (arXiv)
Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: (i) cross-sequence coupling that concentrates probability mass on high-frequency prefixes, (ii) competitive decoding effects that suppress long-tail concepts, and (iii) scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse - divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 19.6-point hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models intended for deployment. Code is available at https://github.com/ulab-uiuc/decompressionlm.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
25 of 37 references.
References · 25
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
- 2020How Context Affects Language Models' Factual Predictionscited by 181
- 2022Locally Typical Samplingcited by 108
- 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
13 more
- 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reasoncited by 47
- 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
- 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
- 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
- 2022Arithmetic Sampling: Parallel Diverse Decoding for Large Language Modelscited by 18
- 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
- 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
- 2025Constrained Sampling for Language Models Should Be Easy: An MCMC Perspectivecited by 14
- 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
- 2025Deep sequence models tend to memorize geometrically; it is unclear whycited by 9
- 2025Efficient and Asymptotically Unbiased Constrained Decoding for Large Language Modelscited by 5
- 2017TAPAS: Two-pass Approximate Adaptive Sampling for Softmaxcited by 3
- 2025Decoding-Free Sampling Strategies for LLM Marginalizationcited by 1
Built on by
Not cited yet.