mlsanityData to 7 Oct 2026

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

arXiv 2407.07093 ↗First posted 9 Jul 2024cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research (Code: https://github.com/LiqunMa/FBI-LLM. Model: https://huggingface.co/LiqunMa/).

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

32 of 38 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
20 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  5. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  7. 2018Model compression via distillation and quantizationcited by 287
  8. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  10. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  11. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  12. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  13. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  14. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  15. 2017WRPN: Wide Reduced-Precision Networkscited by 49
  16. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  17. 2019QKD: Quantization-aware Knowledge Distillationcited by 42
  18. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 22
  19. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 20
  20. 2024Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 3.

12 papers.

  1. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  2. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Modelscited by 10
  3. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 5
  4. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  5. 2025Membership Inference Risks in Quantized Models: A Theoretical and Empirical Studycited by 3
  6. 2026Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMscited by 3
  7. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  8. 2025Achieving binary weight and activation for LLMs using Post-Training Quantizationcited by 1
  9. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Modelscited by 1
  10. 2024Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classificationcited by 0
  11. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0
  12. 2026LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillationcited by 0