mlsanityData to 14 Oct 2026

Ternarization of Vision Language Models for use on edge devices

arXiv 2504.06298 ↗First posted 7 Apr 2025cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

We propose a process to compress a pre-trained Vision Language Model into a ternary version of itself instead of training a ternary model from scratch. A new initialization scheme from pre-trained weights based on the k-means algorithm is proposed to reduce the ternarization time. We implement different custom operators for executing the ternary model on the TensorFlow Lite Engine. We compare the original model with its ternary and binary versions in terms of memory consumption, inference speed and perplexity. We find that the ternary model using our custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 21 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

14 of 21 references.

References · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  8. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  10. 2018Model compression via distillation and quantizationcited by 289
  11. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  12. 20241.58-bit FLUXcited by 31
2 more
  1. 2018Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximationcited by 20
  2. 2018Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracycited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Ternary Quantization: A Survey
  2. 2025TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
  3. 2023Designing strong baselines for ternary neural network quantization through support and mass equalization
  4. 2026TernaryLM: Memory-Efficient Language Modeling via Native 1.5-Bit Quantization with Adaptive Layer-wise Scaling
  5. 2026FTerViT: Fully Ternary Vision Transformer
  6. 2024Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models
  7. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  8. 2026Low-Rank Ternary Adaptation for Fine-Tuning Transformers