mlsanityData to 14 Oct 2026

TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs

arXiv 2504.16266 ↗First posted 22 Apr 2025cs.AR cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Deploying large language models (LLMs) on edge platforms is challenged by their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as little as 1.58 bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected latency of the prefill phase. We present TeLLMe, the first ternary LLM accelerator for low-power FPGAs (e.g., AMD KV260) that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. Our contributions include: (1) a table-lookup matrix engine for ternary matmul that merges grouped activations with online precomputation to minimize resource use; (2) a fused, bandwidth-efficient attention module featuring a reversed reordering scheme to accelerate prefill; and (3) a tightly integrated normalization and quantization--dequantization unit optimized for ultra-low-bit inference. Under a 7W power budget, TeLLMe delivers up to 9 tokens/s throughput over 1,024-token contexts and prefill latencies of 0.55--1.15 s for 64--128 token prompts, marking a significant energy-efficiency advance and establishing a new edge FPGA benchmark for generative AI.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

18 of 20 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  7. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  8. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  9. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  10. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
  11. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 33
  12. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
6 more
  1. 2024Designing Efficient LLM Accelerators for Edge Devicescited by 13
  2. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12
  3. 2024LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAscited by 12
  4. 2022A Two-Stage Efficient 3-D CNN Framework for EEG Based Emotion Recognitioncited by 7
  5. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  6. 2024Table-Lookup MAC: Scalable Processing of Quantised Neural Networks in FPGA Soft Logiccited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edgecited by 6
  2. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  3. 2025TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAscited by 2
  4. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  5. 2025OTARo: Once Tuning for All Precisions toward Robust On-Device LLMscited by 1
  6. 2025PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfigurationcited by 1
  7. 2026Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inferencecited by 1
  8. 2026VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devicescited by 1
  9. 2026Ankhdjet: An Open-Source Compiler for Mask-Programmed Ternary Compute-in-ROM on an Open PDKcited by 0
  10. 2026Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TerEffic: Highly Efficient Ternary LLM Inference on FPGA
  2. 2026VitaLLM: A Versatile, Ultra-Compact Ternary LLM Accelerator with Dependency-Aware Scheduling ↗
  3. 2026TOM: A Ternary Read-only Memory Accelerator for LLM-powered Edge Intelligence
  4. 2025MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs
  5. 2025Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference
  6. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables
  7. 2026PALUTE: Processing-In-Memory Acceleration via Lookup Table for Edge LLM Inference
  8. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs