# SceneMiner: Identity-Preserving Multi-Task Fine-Tuning for Unified BEV Scene Mining

arXiv: https://arxiv.org/abs/2606.11507 · HTML: https://mlsanity.com/paper/2606.11507/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 42 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2504.04348/) (2025)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving](https://mlsanity.com/paper/2310.02251/) (2023)
- [Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.03125/) (2025)

## Abstract (arXiv)

Mining hard, safety-critical scenes from driving logs is bottlenecked by the absence of difficulty labels, and no single proxy, collision risk, trajectory ambiguity, or semantic rarity suffices to find such scenes on its own. We present SceneMiner, a unified, camera-only bird's-eye-view pipeline that emits complementary mining signals from a frozen vision-language backbone in a single forward pass, with no LiDAR or radar: a retrieval embedding for text-prompted scenario search, a multi-label scene-tag distribution, and a continuous physics-based risk score (a motion forecast is a byproduct, not a contribution). Building such a multi-head model exposes our central finding, a failure mode we term cross-task interference: adding or upgrading one head shifts a shared activation stream and degrades weight-frozen sibling heads, so freezing parameters alone is insufficient. Our contribution, identity-preserving multi-task fine-tuning, removes this interference by zero-initializing every new sub-module and freezing every parameter that feeds the shared stream. The mining heads are thereby preserved bit-identically while training only ~102k parameters. The tagging head reaches mAP 0.4614 (micro-F1 0.5557) on 20 scene tags by pooling each scene into 32 visual tokens, and the embedding head supports text-prompted retrieval, validated qualitatively. Code is available at: https://anonymous.4open.science/r/sceneminer_anonymous-64E5