# TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs

arXiv: https://arxiv.org/abs/2504.16266 · HTML: https://mlsanity.com/paper/2504.16266/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [OneBit: Towards Extremely Low-bit Large Language Models](https://mlsanity.com/paper/2402.11295/) (2024)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts](https://mlsanity.com/paper/2305.18691/) (2023)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://mlsanity.com/paper/2407.07093/) (2024)
- [LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs](https://mlsanity.com/paper/2409.11424/) (2024)
- [A Two-Stage Efficient 3-D CNN Framework for EEG Based Emotion Recognition](https://mlsanity.com/paper/2208.00883/) (2022)
- [Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA](https://mlsanity.com/paper/2502.10659/) (2025)
- [Table-Lookup MAC: Scalable Processing of Quantised Neural Networks in FPGA Soft Logic](https://mlsanity.com/paper/2403.11414/) (2024)

## Built on by (most cited first, 10 of 10)

- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization](https://mlsanity.com/paper/2511.13676/) (2025)
- [TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs](https://mlsanity.com/paper/2510.15926/) (2025)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [OTARo: Once Tuning for All Precisions toward Robust On-Device LLMs](https://mlsanity.com/paper/2511.13147/) (2025)
- [PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfiguration](https://mlsanity.com/paper/2512.11550/) (2025)
- [Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference](https://mlsanity.com/paper/2604.25183/) (2026)
- [VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devices](https://mlsanity.com/paper/2605.00320/) (2026)
- [Ankhdjet: An Open-Source Compiler for Mask-Programmed Ternary Compute-in-ROM on an Open PDK](https://mlsanity.com/paper/2608.26206/) (2026)
- [Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions](https://mlsanity.com/paper/2609.27453/) (2026)

## Abstract (arXiv)

Deploying large language models (LLMs) on edge platforms is challenged by their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as little as 1.58 bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected latency of the prefill phase. We present TeLLMe, the first ternary LLM accelerator for low-power FPGAs (e.g., AMD KV260) that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. Our contributions include: (1) a table-lookup matrix engine for ternary matmul that merges grouped activations with online precomputation to minimize resource use; (2) a fused, bandwidth-efficient attention module featuring a reversed reordering scheme to accelerate prefill; and (3) a tightly integrated normalization and quantization--dequantization unit optimized for ultra-low-bit inference. Under a 7W power budget, TeLLMe delivers up to 9 tokens/s throughput over 1,024-token contexts and prefill latencies of 0.55--1.15 s for 64--128 token prompts, marking a significant energy-efficiency advance and establishing a new edge FPGA benchmark for generative AI.