# RLM: A Vision-Language Model Approach for Radar Scene Understanding

arXiv: https://arxiv.org/abs/2511.21105 · HTML: https://mlsanity.com/paper/2511.21105/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Open-Vocabulary Universal Image Segmentation with MaskCLIP](https://mlsanity.com/paper/2208.08984/) (2022)
- [K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions](https://mlsanity.com/paper/2206.08171/) (2022)
- [RADIATE: A Radar Dataset for Automotive Perception in Bad Weather](https://mlsanity.com/paper/2010.09076/) (2020)
- [SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger](https://mlsanity.com/paper/2303.17561/) (2023)
- [RADDet: Range-Azimuth-Doppler based Radar Object Detection for Dynamic Road Users](https://mlsanity.com/paper/2105.00363/) (2021)
- [Hidden Gems: 4D Radar Scene Flow Learning Using Cross-Modal Supervision](https://mlsanity.com/paper/2303.00462/) (2023)
- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)

## Built on by (most cited first, 2 of 2)

- [Weather-Robust Scene Semantics with Vision-Aligned 4D Radar](https://mlsanity.com/paper/2605.07367/) (2026)
- [Radar4D-VLM: Proposal-Grounded Temporal 4D Radar Reasoning Across Frozen Language Models](https://mlsanity.com/paper/2608.04130/) (2026)

## Abstract (arXiv)

Radar sensors provide reliable perception across adverse weather, lighting, and long-range conditions, yet existing machine learning approaches remain fragmented and task-specific, with each downstream task employing distinct architectures and training objectives. We present RadarVLM, a vision-language framework that learns unified scene-level representations through structured spatial language supervision. Leveraging the CARLA simulator with a realistic radar model, we collect over 800k radar-caption pairs across 110+ hours of simulated driving in diverse scenarios. We make two key contributions: (1) a structured caption framework encoding vehicle distributions in the radar's native coordinate system, and (2) Spatially-Grounded CLIP (SG-CLIP) objective that replaces binary matching with continuous scene similarity, enabling fine-grained spatial reasoning. We further propose localization-aware evaluation metrics that directly assess spatial accuracy beyond traditional linguistic similarity measures. Validated on generative captioning and vehicle segmentation, SG-CLIP achieves up to 50% relative F1-score improvement over vanilla CLIP and a 21% AP gain on segmentation, demonstrating that language grounding produces spatially structured representations.