Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp
Not on the map. Why?
Abstract (arXiv)
This article optimizes the inference performance of the Qwen-1.8B model by performing Int8 quantization, vectorizing some operators in llama.cpp, and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 2 of its 8 references that are in the map. First posted on arXiv 16 Jun 2024.
Builds on
2 of 8 references.
References · 2
- 2023Qwen Technical Reportcited by 3,344
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
- 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
- 2023Efficient LLM Inference on CPUs
- 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
- 2025Optimizing LLMs Using Quantization for Mobile Execution
- 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
- 2025F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs
- 2024LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs