# Leveraging Multimodal-LLMs Assisted by Instance Segmentation for Intelligent Traffic Monitoring

arXiv: https://arxiv.org/abs/2502.11304 · HTML: https://mlsanity.com/paper/2502.11304/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObility](https://mlsanity.com/paper/2409.09040/) (2024)
- [iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvement](https://mlsanity.com/paper/2407.06025/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis](https://mlsanity.com/paper/2501.10604/) (2025)
- [TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning](https://mlsanity.com/paper/2404.09275/) (2024)
- [Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm](https://mlsanity.com/paper/2411.10869/) (2024)
- [The Role of LLMs in Sustainable Smart Cities: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2402.14596/) (2024)
- [Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events](https://mlsanity.com/paper/2406.13894/) (2024)
- [Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks](https://mlsanity.com/paper/2412.01132/) (2024)
- [Multimodal LLM for Intelligent Transportation Systems](https://mlsanity.com/paper/2412.11683/) (2024)

## Built on by (most cited first, 5 of 5)

- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025)
- [Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis](https://mlsanity.com/paper/2510.11907/) (2025)
- [Spatiotemporal Semantic V2X Framework for Cooperative Collision Prediction](https://mlsanity.com/paper/2601.17216/) (2026)
- [A Lightweight Digital-Twin-Based Framework for Edge-Assisted Vehicle Tracking and Collision Prediction](https://mlsanity.com/paper/2603.07338/) (2026)

## Abstract (arXiv)

A robust and efficient traffic monitoring system is essential for smart cities and Intelligent Transportation Systems (ITS), using sensors and cameras to track vehicle movements, optimize traffic flow, reduce congestion, enhance road safety, and enable real-time adaptive traffic control. Traffic monitoring models must comprehensively understand dynamic urban conditions and provide an intuitive user interface for effective management. This research leverages the LLaVA visual grounding multimodal large language model (LLM) for traffic monitoring tasks on the real-time Quanser Interactive Lab simulation platform, covering scenarios like intersections, congestion, and collisions. Cameras placed at multiple urban locations collect real-time images from the simulation, which are fed into the LLaVA model with queries for analysis. An instance segmentation model integrated into the cameras highlights key elements such as vehicles and pedestrians, enhancing training and throughput. The system achieves 84.3% accuracy in recognizing vehicle locations and 76.4% in determining steering direction, outperforming traditional models.