# PaveCap: The First Multimodal Framework for Comprehensive Pavement Condition Assessment with Dense Captioning and PCI Estimation

arXiv: https://arxiv.org/abs/2408.04110 · HTML: https://mlsanity.com/paper/2408.04110/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 32 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [CPTR: Full Transformer Network for Image Captioning](https://mlsanity.com/paper/2101.10804/) (2021)
- [Pavement Image Datasets: A New Benchmark Dataset to Classify and Densify Pavement Distresses](https://mlsanity.com/paper/1910.11123/) (2019)
- [Deep Machine Learning Approach to Develop a New Asphalt Pavement Condition Index](https://mlsanity.com/paper/2004.13314/) (2020)
- [Image2PCI -- A Multitask Learning Framework for Estimating Pavement Condition Indices Directly from Images](https://mlsanity.com/paper/2310.08538/) (2023)

## Built on by (most cited first, 6 of 6)

- [Integrating Travel Behavior Forecasting and Generative Modeling for Predicting Future Urban Mobility and Spatial Transformations](https://mlsanity.com/paper/2503.21158/) (2025)
- [An Improved ResNet50 Model for Predicting Pavement Condition Index (PCI) Directly from Pavement Images](https://mlsanity.com/paper/2504.18490/) (2025)
- [Demographics-Informed Neural Network for Multi-Modal Spatiotemporal forecasting of Urban Growth and Travel Patterns Using Satellite Imagery](https://mlsanity.com/paper/2506.12456/) (2025)
- [Prompt-Guided Spatial Understanding with RGB-D Transformers for Fine-Grained Object Relation Reasoning](https://mlsanity.com/paper/2510.11996/) (2025)
- [PaveSync: A Unified and Comprehensive Dataset for Pavement Distress Analysis and Classification](https://mlsanity.com/paper/2512.20011/) (2025)
- [Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment](https://mlsanity.com/paper/2604.08212/) (2026)

## Abstract (arXiv)

This research introduces the first multimodal approach for pavement condition assessment, providing both quantitative Pavement Condition Index (PCI) predictions and qualitative descriptions. We introduce PaveCap, a novel framework for automated pavement condition assessment. The framework consists of two main parts: a Single-Shot PCI Estimation Network and a Dense Captioning Network. The PCI Estimation Network uses YOLOv8 for object detection, the Segment Anything Model (SAM) for zero-shot segmentation, and a four-layer convolutional neural network to predict PCI. The Dense Captioning Network uses a YOLOv8 backbone, a Transformer encoder-decoder architecture, and a convolutional feed-forward module to generate detailed descriptions of pavement conditions. To train and evaluate these networks, we developed a pavement dataset with bounding box annotations, textual annotations, and PCI values. The results of our PCI Estimation Network showed a strong positive correlation (0.70) between predicted and actual PCIs, demonstrating its effectiveness in automating condition assessment. Also, the Dense Captioning Network produced accurate pavement condition descriptions, evidenced by high BLEU (0.7445), GLEU (0.5893), and METEOR (0.7252) scores. Additionally, the dense captioning model handled complex scenarios well, even correcting some errors in the ground truth data. The framework developed here can greatly improve infrastructure management and decision18 making in pavement maintenance.