mlsanityData to 14 Oct 2026

LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment

arXiv 2603.00042 ↗First posted 9 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We identify the Spectral Energy Gain in extreme model compression, where low-rank binary approximations outperform tiny-rank floating-point baselines for heavy-tailed spectra. However, prior attempts fail to realize this potential, trailing state-of-the-art 1-bit methods. We attribute this degradation to Latent Geometry Misalignment: standard singular vectors exhibit high coherence (spiky distribution), the worst-case geometry for binary quantization. To realize this gain, we propose LittleBit-2, a framework employing Internal Latent Rotation and Joint Iterative Quantization (Joint-ITQ). This approach acts as a geometric preconditioner, aligning coherent latent distributions with the binary hypercube with zero inference overhead. Empirically, LittleBit-2 establishes a new state-of-the-art in the sub-1-bit regime (1~0.1 bpp) on Llama-2 and Llama-3, matching the fidelity of leading 1-bit baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

27 of 31 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
15 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  3. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  5. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  6. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  7. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  8. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  9. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  10. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  11. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  12. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  13. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  14. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  15. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization
  2. 2025SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
  3. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
  4. 2026HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
  5. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
  6. 2026Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization
  7. 2026Few Bits, One Law: Toward W2A4KV2
  8. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Design