# F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs

arXiv: https://arxiv.org/abs/2510.13401 · HTML: https://mlsanity.com/paper/2510.13401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [NPE: An FPGA-based Overlay Processor for Natural Language Processing](https://mlsanity.com/paper/2104.06535/) (2021)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Support](https://mlsanity.com/paper/2203.06673/) (2022)

## Abstract (arXiv)

Large Language Models (LLMs) have become increasingly prominent for daily tasks, from improving sound-totext translation to generating additional frames for the latest video games. With the help of LLM inference frameworks, such as llama.cpp, which support optimizations such as KV-caching and quantization, it is now easier than ever to deploy LLMs on edge devices. Quantization is fundamental to enable LLMs on resource-constrained edge devices, and llama.cpp utilizes block floating point (BFP) quantization to drastically reduce the bit width of weights and input tensors, the memory footprint, and the computational power required to run LLMs. LLMs are typically quantized with mixed BFP quantization across the model layers to reduce the loss of model accuracy due to quantization. Therefore, to efficiently accelerate across the layers of BFP-quantized LLMs, specialized accelerators need to support different BFP variants without reconfiguration. To address this issue, we propose a Flexible Block FloatingPoint Quantization (F-BFQ) accelerator, which can dynamically switch between two BFP quantization variants and perform matrix multiplication (MatMul) operations. Our initial F-BFQ accelerator design, deployed on the AMD Kria board, reduces inference time by 1.4x on average over the Arm NEON-based CPU execution across three BFP quantized LLMs while achieving 5.2 tokens per second (~3.9 words per second).