# Taming Vision Priors for Data Efficient mmWave Channel Modeling

arXiv: https://arxiv.org/abs/2603.13383 · HTML: https://mlsanity.com/paper/2603.13383/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views](https://mlsanity.com/paper/2404.03650/) (2024)
- [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023)
- [Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing](https://mlsanity.com/paper/2404.01223/) (2024)
- [NeWRF: A Deep Learning Framework for Wireless Radiation Field Reconstruction and Channel Prediction](https://mlsanity.com/paper/2403.03241/) (2024)
- [Learning Radio Environments by Differentiable Ray Tracing](https://mlsanity.com/paper/2311.18558/) (2023)
- [PMNet: Robust Pathloss Map Prediction via Supervised Learning](https://mlsanity.com/paper/2211.10527/) (2022)
- [How well does CLIP understand texture?](https://mlsanity.com/paper/2203.11449/) (2022)
- [Two beams are better than one: Enabling reliable and high throughput mmWave links](https://mlsanity.com/paper/2101.04249/) (2021)

## Built on by (most cited first, 2 of 2)

- [Point2Radio: A Foundation Model for Cross-Scene Radio Fields from Material-Aware Point Clouds](https://mlsanity.com/paper/2607.28994/) (2026)
- [KODAMA: Multimodal Digital Twin Reconstruction for Urban RF Propagation Modelling](https://mlsanity.com/paper/2609.07298/) (2026)

## Abstract (arXiv)

Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10× while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.