mlsanityData to 14 Oct 2026

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

arXiv 2607.02909 ↗First posted 3 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization (HiR2), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that HiR2 effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2ICML2026.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 73 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

60 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models28 Feb
  2. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning7 Feb
  3. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation21 Apr
  4. 2025Taxonomy-Aware Evaluation of Vision-Language Models7 Apr
  5. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models25 Jan
  6. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities21 Dec
  7. 2024Cross-modal Information Flow in Multimodal Large Language Models27 Nov
  8. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification20 Jun

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2020Supervised Contrastive Learningcited by 2,829
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  7. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  8. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  9. 2018Hyperbolic Neural Networkscited by 402
  10. 2018Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometrycited by 265
  11. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  12. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
38 more
  1. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  2. 2023Hyperbolic Image-Text Representationscited by 136
  3. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  4. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  5. 2019Mixed-curvature Variational Autoencoderscited by 86
  6. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  7. 2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 74
  8. 2021Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifierscited by 57
  9. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  10. 2013Evaluation Measures for Hierarchical Classification: a unified view and novel approachescited by 46
  11. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  12. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  13. 2022Label Relation Graphs Enhanced Hierarchical Residual Network for Hierarchical Multi-Granularity Classificationcited by 29
  14. 2024Language Models as Hierarchy Encoderscited by 28
  15. 2025HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Expertscited by 26
  16. 2023Hyperbolic Geometric Graph Representation Learning for Hierarchy-imbalance Node Classificationcited by 23
  17. 2025Aligning Text to Image in Diffusion Models is Easier Than You Thinkcited by 22
  18. 2022Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classificationcited by 21
  19. 2023HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understandingcited by 21
  20. 2025KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Groundingcited by 21
  21. 2024Hyperbolic Geometric Latent Diffusion Model for Graph Generationcited by 20
  22. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20
  23. 2023Hyperbolic Active Learning for Semantic Segmentation under Domain Shiftcited by 18
  24. 2024Lorentzian Residual Neural Networkscited by 17
  25. 2024Visually Consistent Hierarchical Image Classificationcited by 16
  26. 2024Learning Structured Representations with Hyperbolic Embeddingscited by 16
  27. 2022Curved Geometric Networks for Visual Anomaly Recognitioncited by 15
  28. 2023Exploring Data Geometry for Continual Learningcited by 15
  29. 2023ProTeCt: Prompt Tuning for Taxonomic Open Set Classificationcited by 12
  30. 2025SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Modelscited by 11
  31. 2024Hyperbolic Learning with Multimodal Large Language Modelscited by 10
  32. 2024Hyperbolic-constraint Point Cloud Reconstruction from Single RGB-D Imagescited by 8
  33. 2025On Large Multimodal Models as Open-World Image Classifierscited by 8
  34. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  35. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  36. 2025Curvature Learning for Generalization of Hyperbolic Neural Networkscited by 4
  37. 2024Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transportcited by 3
  38. 2025Modality Alignment across Trees on Heterogeneous Hyperbolic Manifoldscited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchy-Aware Fine-Tuning of Vision-Language Models
  2. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoning
  3. 2026Cross-Modal Taxonomic Generalization in (Vision-) Language Models
  4. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representations
  5. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  6. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  7. 2026ARGENT: Adaptive Hierarchical Image-Text Representations
  8. 2025Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders