mlsanityData to 14 Oct 2026

Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models

arXiv 2512.21337 ↗First posted 24 Dec 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We expose a significant popularity bias in state-of-the-art vision-language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open benchmark for this task: the YearGuessr dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 80 references that are in the map. First posted on arXiv 24 Dec 2025.

Builds on

50 of 80 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning12 Jul
  2. 2023CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model9 Apr

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  11. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  12. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
36 more
  1. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  2. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  3. 2023GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localizationcited by 187
  4. 2022TransGeo: Transformer Is All You Need for Cross-view Image Geo-localizationcited by 168
  5. 2022SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understandingcited by 167
  6. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  7. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  8. 2019Rank consistent ordinal regression for neural networks with application to age estimationcited by 133
  9. 2016PlaNet - Photo Geolocation with Convolutional Neural Networkscited by 115
  10. 2017Revisiting IM2GPS in the Deep Learning Eracited by 80
  11. 2014Material Recognition in the Wild with the Materials in Context Databasecited by 79
  12. 2022Can Language Understand Depth?cited by 69
  13. 2018CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Mapscited by 68
  14. 2021Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regressioncited by 58
  15. 2022Where in the World is this Image? Transformer-based Geo-localization in the Wildcited by 58
  16. 2023PIGEON: Predicting Image Geolocationscited by 56
  17. 2023Where We Are and What We're Looking At: Query Based Worldwide Image Geo-localization Using Hierarchies and Scenescited by 55
  18. 2022Moving Window Regression: A Novel Approach to Ordinal Regressioncited by 44
  19. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  20. 2023SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understandingcited by 38
  21. 2024Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentationcited by 38
  22. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  23. 2024LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wildcited by 21
  24. 2024AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localizationcited by 21
  25. 2024Teach CLIP to Develop a Number Sense for Ordinal Regressioncited by 20
  26. 2025See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigationcited by 18
  27. 2023EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadatacited by 17
  28. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 17
  29. 2022G^3: Geolocation via Guidebook Groundingcited by 12
  30. 2024Zero-shot Building Age Classification from Facade Image Using GPT-4cited by 8
  31. 2025OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotationscited by 7
  32. 2018Automatic Prediction of Building Age from Photographscited by 5
  33. 2021WikiChurches: A Fine-Grained Dataset of Architectural Styles with Real-World Challengescited by 3
  34. 2022MTBF-33: A multi-temporal building footprint dataset for 33 counties in the United States (1900-2015)cited by 2
  35. 2025Building Age Estimation: A New Multi-Modal Benchmark Dataset and Community Challengecited by 2
  36. 2024CMAB: A First National-Scale Multi-Attribute Building Dataset in China Derived from Open Source Data and GeoAIcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026One Figure, Every Canvas: Editable Flowchart Relayout via Agentic Pipelinecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2026AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
  3. 2025Vision Language Models are Biased
  4. 2026Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models
  5. 2025CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models
  6. 2026Leveraging Multimodal LLMs for Built Environment and Housing Attribute Assessment from Street-View Imagery
  7. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  8. 2026Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated