mlsanityData to 14 Oct 2026

LaMoC: Loss-Aware Modular Compression for LLMs

arXiv 2608.30226 ↗First posted 31 Aug 2026cs.AI cs.CL cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

17 of 30 references.

References · 17

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  3. 2019Importance Estimation for Neural Network Pruningcited by 541
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  7. 2022Language model compression with weighted low-rank factorizationcited by 195
  8. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  9. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  10. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  11. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17
  12. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
5 more
  1. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8
  2. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 7
  3. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  4. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  5. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2

Built on by

Not cited yet.