# A Vision-Language Framework for Measuring Social Life on Sidewalks

arXiv: https://arxiv.org/abs/2609.28476 · HTML: https://mlsanity.com/paper/2609.28476/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2312.02284/) (2023)
- [V-IRL: Grounding Virtual Intelligence in Real Life](https://mlsanity.com/paper/2402.03310/) (2024)
- [EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignment](https://mlsanity.com/paper/2312.08548/) (2023)
- [Estimating city-level travel patterns using street imagery: a case study of using Google Street View in Britain](https://mlsanity.com/paper/1802.02915/) (2018)
- [MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenes](https://mlsanity.com/paper/2509.13484/) (2025)
- [Computer Vision for Transit Travel Time Prediction: An End-to-End Framework Using Roadside Urban Imagery](https://mlsanity.com/paper/2211.12322/) (2022)

## Abstract (arXiv)

While a number of methods exist for counting pedestrians in street-view imagery, these mostly ignore the social dimensions of pedestrian activity. A street traversed by a high volume of pedestrians has the same headcount as a street where people linger, sit, and socialize. This paper presents a vision-language framework for extracting social indicators from street-level imagery. Panoramic street-level imagery is reprojected to sidewalk-facing sideviews with preserved timestamps. A vision-language model (VLM)-based activity detection system codes each person across ten independent observable dimensions, resolving a systematic failure mode in which models prompted with high-level social categories conflate observable states with contextual inferences. The resulting social indicator system produces a Social Dwelling Index (SDI) that jointly considers pedestrian grouping and dwelling, provides activity labels documenting behavioral diversity, and issues binary flags for the presence of accessibility-sensitive populations. We apply the framework to 102,514 sideviews in New York City, revealing that pedestrian volume and SDI are only weakly associated (r = 0.168): streets with the highest foot traffic are not where social activity is most intense. The framework provides a scalable method for measuring not only how many people are on city sidewalks, but also their grouping, posture, and activity type, summarizing the non-transient activities that occur on city sidewalks.