mlsanityData to 14 Oct 2026

Motion-Compensated Weight Compression

arXiv 2605.24754 ↗First posted 23 May 2026cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural network weights are increasingly a bottleneck for deployment, yet most compression pipelines treat layers independently and overlook cross-layer redundancy induced by function-preserving symmetries. We propose Motion-Compensated Weight Compression (MCWC), a weight-only codec that aligns permutation-symmetric blocks (e.g., hidden units and attention heads) to maximize cross-layer correspondence, turning depth into a predictable sequence. In the aligned coordinate system, MCWC uses a lightweight layer-sequential predictor with periodic keyframes and encodes only quantized prediction residuals using a learned entropy model trained under a rate distortion objective. A simple decoder reconstructs deployable weights by entropy decoding, dequantization, predictor-driven reconstruction, and inverse alignment, enabling fast weight materialization for inference. Across Transformer language modeling and vision classification, MCWC improves the rate accuracy Pareto frontier over strong quantization and learned weight-codec baselines, while maintaining competitive decode time. Ablations confirm that alignment, prediction, entropy modeling, and keyframe scheduling are each necessary for the full gains. Our code is available via https://github.com/Ism-ail11/MCWC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

43 of 58 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2023Mistral 7Bcited by 3,164
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
31 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2024Mixtral of Expertscited by 1,823
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2018Variational image compression with a scale hyperpriorcited by 935
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  13. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  14. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  15. 2018Joint Autoregressive and Hierarchical Priors for Learned Image Compressioncited by 664
  16. 2019Learned Step Size Quantizationcited by 628
  17. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  18. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  19. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  20. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  21. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 332
  22. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  23. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  24. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  25. 2020Talking-Heads Attentioncited by 62
  26. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  27. 2018Universal Deep Neural Network Compressioncited by 19
  28. 2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learningcited by 6
  29. 2025EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devicescited by 2
  30. 2024Getting Free Bits Back from Rotational Symmetries in LLMscited by 1
  31. 2025Neural Weight Compression for Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates
  2. 2021Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression
  3. 2021Compact representations of convolutional neural networks via weight pruning and quantization
  4. 2026WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs
  5. 2026CurveCodec 2: Skeleton-agnostic animation compression with a learned entropy model
  6. 2026Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
  7. 2022Quantized Sparse Weight Decomposition for Neural Network Compression
  8. 2025SineLoRADelta: Sine-Activated Delta Compression