# MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance

arXiv: https://arxiv.org/abs/2509.09730 · HTML: https://mlsanity.com/paper/2509.09730/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 65 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 30 of 32)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking](https://mlsanity.com/paper/1511.04136/) (2015)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by A Benchmark](https://mlsanity.com/paper/1904.12634/) (2019)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [Exploring Effective Factors for Improving Visual In-Context Learning](https://mlsanity.com/paper/2304.04748/) (2023)
- [Abductive Ego-View Accident Video Understanding for Safe Driving Perception](https://mlsanity.com/paper/2403.00436/) (2024)
- [CADP: A Novel Dataset for CCTV Traffic Camera based Accident Analysis](https://mlsanity.com/paper/1809.05782/) (2018)
- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [Reading Between the Lanes: Text VideoQA on the Road](https://mlsanity.com/paper/2307.03948/) (2023)
- [Context Disentangling and Prototype Inheriting for Robust Visual Grounding](https://mlsanity.com/paper/2312.11967/) (2023)
- [TAD: A Large-Scale Benchmark for Traffic Accidents Detection from Video Surveillance](https://mlsanity.com/paper/2209.12386/) (2022)
- [Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline](https://mlsanity.com/paper/2408.03120/) (2024)
- [Visual Position Prompt for MLLM based Visual Grounding](https://mlsanity.com/paper/2503.15426/) (2025)
- [A Multimodal Benchmark Dataset and Model for Crop Disease Diagnosis](https://mlsanity.com/paper/2503.06973/) (2025)
- [RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives](https://mlsanity.com/paper/2503.21459/) (2025)
- [A Study of Situational Reasoning for Traffic Understanding](https://mlsanity.com/paper/2306.02520/) (2023)
- [Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement](https://mlsanity.com/paper/2303.08983/) (2023)

## Built on by (most cited first, 1 of 1)

- [Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges](https://mlsanity.com/paper/2608.23845/) (2026)

## Abstract (arXiv)

General-domain large multimodal models (LMMs) have achieved significant advances in various image-text tasks. However, their performance in the Intelligent Traffic Surveillance (ITS) domain remains limited due to the absence of dedicated multimodal datasets. To address this gap, we introduce MITS (Multimodal Intelligent Traffic Surveillance), the first large-scale multimodal benchmark dataset specifically designed for ITS. MITS includes 170,400 independently collected real-world ITS images sourced from traffic surveillance cameras, annotated with eight main categories and 24 subcategories of ITS-specific objects and events under diverse environmental conditions. Additionally, through a systematic data generation pipeline, we generate high-quality image captions and 5 million instruction-following visual question-answer pairs, addressing five critical ITS tasks: object and event recognition, object counting, object localization, background analysis, and event reasoning. To demonstrate MITS's effectiveness, we fine-tune mainstream LMMs on this dataset, enabling the development of ITS-specific applications. Experimental results show that MITS significantly improves LMM performance in ITS applications, increasing LLaVA-1.5's performance from 0.494 to 0.905 (+83.2%), LLaVA-1.6's from 0.678 to 0.921 (+35.8%), Qwen2-VL's from 0.584 to 0.926 (+58.6%), and Qwen2.5-VL's from 0.732 to 0.930 (+27.0%). We release the dataset, code, and models as open-source, providing high-value resources to advance both ITS and LMM research.