mlsanityData to 14 Oct 2026

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

arXiv 2602.19442 ↗First posted 23 Feb 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 62 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

35 of 62 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2020Concept Bottleneck Modelscited by 1,032
  11. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
23 more
  1. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  3. 2022Post-hoc Concept Bottleneck Modelscited by 293
  4. 2023Label-Free Concept Bottleneck Modelscited by 285
  5. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  6. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  8. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  9. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  10. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  11. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 81
  12. 2023UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Webcited by 78
  13. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  14. 2016Deep Learning the City : Quantifying Urban Perception At A Global Scalecited by 65
  15. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  16. 2024Stochastic Concept Bottleneck Modelscited by 51
  17. 2024Concept Bottleneck Large Language Modelscited by 43
  18. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  19. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  20. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 12
  21. 2025Global urban visual perception varies across demographics and personalitiescited by 10
  22. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  23. 2025Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmarkcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Culturally uneven urban perception in large language modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
  2. 2024Self-Supervised Visual Preference Alignment
  3. 2026A Dataset for Dynamic Human Preferences for Vision Language Models
  4. 2026Semantic Misalignment in Vision-Language Models under Perceptual Degradation
  5. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  6. 2025LVLM-Aided Alignment of Task-Specific Vision Models
  7. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  8. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences