mlsanityData to 7 Oct 2026

Hierarchy-Aware Fine-Tuning of Vision-Language Models

arXiv 2512.21529 ↗First posted 25 Dec 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 43 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

29 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
15 more
  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  2. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  3. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  4. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  5. 2023UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modelingcited by 40
  6. 2017B-CNN: Branch Convolutional Neural Network for Hierarchical Classificationcited by 37
  7. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  8. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  9. 2018Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embeddingcited by 28
  10. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  11. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  12. 2022Hierarchical classification at multiple operating pointscited by 14
  13. 2020Hierarchical Deep Convolutional Neural Networks for Multi-category Diagnosis of Gastrointestinal Disorders on Histopathological Imagescited by 5
  14. 2021MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classificationcited by 5
  15. 2024Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
  2. 2025Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning
  3. 2026Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
  4. 2026Hierarchical Compression of Vision-Language Model Benchmarks
  5. 2025Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
  6. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  7. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
  8. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models