mlsanityData to 7 Oct 2026

RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies

arXiv 2608.28656 ↗First posted 21 Aug 2026cs.RO cs.CVcited by 0

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Behavior-cloned Vision-Language-Action (VLA) driving policies struggle with rare rule-governed maneuvers at signalized intersections. Braking and launching examples contribute little to averaged trajectory loss, while fused representations lack explicit supervision for the governing traffic-light and stop-line state. We present RedLight-VLA, a training objective that uses expert futures and automatically generated perception targets without additional manual rule annotation. First, trajectory-derived behavioral reweighting (BR) emphasizes rare deceleration and acceleration using rotation-invariant longitudinal dynamics and a scale-preserving reduction that exactly recovers the baseline when disabled. Second, parallel auxiliary (AUX) heads ground traffic-light and stop-line state in continuous post-fusion rule tokens, without autoregressive language generation or changes to the trajectory decoder. We evaluate on a curated set of 20 s sequences with a 5 s prediction horizon. Controlled variants share the same backbone, training data, decoder, and evaluation population. Against an otherwise identical VLA baseline, RedLight-VLA reduces red-light stop-line overshoot from 7.3% to6.8%, reduces stop-line velocity error by 12.7%, and improves 3 s trafficlight-sliced ADE/FDE from 0.274/0.964 m to 0.247/0.897 m. Green-light false stops increase from 3.2% to 3.9%; however, combining BR with AUX supervision mitigates the larger increase observed for AUX alone (4.0%). The combined model also improves non-traffic-light ADE/FDE from 0.268/0.956 m to 0.241/0.876 m and outperforms either mechanism alone on all four sliced displacement measures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 38 of its 47 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

38 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail30 Oct
  2. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation25 Mar
  3. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment12 Mar
  4. 2024EMMA: End-to-End Multimodal Model for Autonomous Driving30 Oct

Other references · 34

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  6. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 2020Concept Bottleneck Modelscited by 1,022
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
22 more
  1. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  2. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  3. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  4. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  8. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  12. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  13. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  14. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  15. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  16. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  17. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  18. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  19. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  20. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  21. 2019Challenges in Designing Datasets and Validation for Autonomous Drivingcited by 23
  22. 2022D2-TPred: Discontinuous Dependency for Trajectory Prediction under Traffic Lightscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VLALight: A Vision-Language-Action Model for Traffic Signal Control
  2. 2025VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture
  3. 2026Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
  4. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  5. 2026RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
  6. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  7. 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
  8. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future