mlsanityData to 14 Oct 2026

EdgeFM: Efficient Edge Inference for Vision-Language Models

arXiv 2604.27476 ↗First posted 30 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision-language models (VLMs) have demonstrated strong applicability in edge industrial applications, yet their deployment remains severely constrained by requirements for deterministic low latency and stable execution under resource limitations. Existing frameworks either rely on bloated general-purpose designs or force developers into opaque, hardware-specific closed-source ecosystems, leading to hardware lock-in limitation and poor cross-platform adaptability. Observing that modern AI agents can efficiently search and tune configurations to generate highly optimized low-level kernels for standard LLM operators, we propose EdgeFM, a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment. EdgeFM removes non-essential features to reduce single-request latency, and encapsulates agent-tuned kernel optimizations as a modular library of reusable skills. By allowing direct invocation of these skills rather than waiting for closed-source implementations, it effectively closes the performance gap long dominated by proprietary toolchains. The framework natively supports mainstream platforms including x86 and NVIDIA Orin SoCs, and represents the first end-to-end VLA deployment on the domestic Horizon Journey platform, enhancing cross-platform portability. In most cases, it yields clearly better inference performance than conventional vendor-specific toolchains, achieving up to 1.49 times speedup over TensorRT-Edge-LLM on the NVIDIA Orin platform. Experimental results show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 39 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

24 of 39 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
12 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  8. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  9. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  10. 2026Qwen3-TTS Technical Reportcited by 118
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  12. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
  2. 2026Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design
  3. 2026Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment
  4. 2026AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference
  5. 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
  6. 2026Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference
  7. 2025Vision-Language Models for Edge Networks: A Comprehensive Survey
  8. 2026LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization