mlsanityData to 14 Oct 2026

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

arXiv 2609.31564 ↗First posted 25 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator. Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 finetuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points. The trend extends to different grammar compressors (LZ78, SEQUITUR), over which the networks has not be finetuned against. To our knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

18 of 27 references.

References · 18

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  9. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  11. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  12. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
6 more
  1. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  2. 2025TorchAO: PyTorch-Native Training-to-Serving Model Optimizationcited by 26
  3. 2022Improving Matrix-vector Multiplication via Lossless Grammar-Compressed Matricescited by 4
  4. 2023Self-Compressing Neural Networkscited by 2
  5. 2026Algorithmic Simplification of Neural Networks with Mosaic-of-Motifscited by 2
  6. 2026Zero-Shot Quantization via Weight-Space Arithmeticcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Neural Weight Compression for Language Models
  2. 2024Training Neural Networks as Recognizers of Formal Languages
  3. 2026LegoLM: Structured Weight Sharing for Large Language Models
  4. 2026Rethinking Weight Tying: Pseudo-Inverse Tying for LM Stable Training and Updates
  5. 2024Data-free Weight Compress and Denoise for Large Language Models
  6. 2025Text2Weight: Bridging Natural Language and Neural Network Weight Spaces
  7. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
  8. 2026Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data