mlsanityData to 7 Oct 2026

VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

arXiv 2111.12083 ↗First posted 23 Nov 2021cs.RO cs.CV cs.LGcited by 81

Not on the map. Why?

Cited by 32 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

40 of 51 references.

References · 40

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2016End to End Learning for Self-Driving Carscited by 1,000
  5. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  6. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  7. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  8. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  10. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2019Learning by Cheatingcited by 361
28 more
  1. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  2. 2020V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Predictioncited by 292
  3. 2016The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAMcited by 203
  4. 2018Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Carscited by 197
  5. 2019Events-to-Video: Bringing Modern Computer Vision to Event Camerascited by 191
  6. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  7. 2019Video to Events: Recycling Video Datasets for Event Camerascited by 162
  8. 2021TimeLens: Event-based Video Frame Interpolationcited by 150
  9. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  10. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  11. 2020Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancycited by 125
  12. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  13. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  14. 2019Depth Completion from Sparse LiDAR Data with Depth-Normal Constraintscited by 108
  15. 2021Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Predictioncited by 88
  16. 2019Urban Driving with Conditional Imitation Learningcited by 84
  17. 2020Learning Monocular Dense Depth from Eventscited by 84
  18. 2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 79
  19. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  20. 2020Flightmare: A Flexible Quadrotor Simulatorcited by 69
  21. 2018On Offline Evaluation of Vision-based Driving Modelscited by 62
  22. 2020Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learningcited by 60
  23. 2016Event-based, 6-DOF Camera Tracking from Photometric Depth Mapscited by 54
  24. 2018Variational End-to-End Navigation and Localizationcited by 34
  25. 2017Sim4CV: A Photo-Realistic Simulator for Computer Vision Applicationscited by 31
  26. 2021A Survey on Simulators for Testing Self-Driving Carscited by 26
  27. 2021Efficient and Robust LiDAR-Based End-to-End Navigationcited by 22
  28. 2021A Surface Geometry Model for LiDAR Depth Completioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 23; 2024: 22; 2025: 17; 2026: 11.

81 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  3. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  4. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  5. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  6. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  7. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 91
  8. 2024Towards Realistic Scene Generation with LiDAR Diffusion Modelscited by 67
  9. 2024Recent Event Camera Innovations: A Surveycited by 67
  10. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 65
  11. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  12. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
38 more
  1. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  2. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  3. 2024GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPScited by 43
  4. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  6. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  7. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  8. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  9. 2023Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Drivingcited by 26
  10. 2024Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Videocited by 26
  11. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  12. 2023Advancements in Point Cloud Data Augmentation for Deep Learning: A Surveycited by 24
  13. 2023LightSim: Neural Lighting Simulation for Urban Scenescited by 20
  14. 2024Probing Multimodal LLMs as World Models for Drivingcited by 19
  15. 2023Implicit Event-RGBD Neural SLAMcited by 16
  16. 2024HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenescited by 16
  17. 2024VDG: Vision-Only Dynamic Gaussian for Driving Simulationcited by 16
  18. 2023MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulationcited by 15
  19. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  20. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  21. 2024Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networkscited by 12
  22. 2022Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Drivingcited by 10
  23. 2024OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Drivingcited by 10
  24. 2023DGNR: Density-Guided Neural Point Rendering of Large Driving Scenescited by 9
  25. 2024XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesiscited by 9
  26. 2023Event-Free Moving Object Segmentation from Moving Ego Vehiclecited by 8
  27. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  28. 2022Deep Learning based Computer Vision Methods for Complex Traffic Environments Perception: A Reviewcited by 7
  29. 2025Event Quality Score (EQS): Assessing the Realism of Simulated Event Camera Streams via Distances in Latent Spacecited by 7
  30. 2022Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learningcited by 6
  31. 2023From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Drivingcited by 6
  32. 2022Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gapcited by 5
  33. 2023Learning Stability Attention in Vision-based End-to-end Driving Policiescited by 5
  34. 2023Learning with Chemical versus Electrical Synapses -- Does it Make a Difference?cited by 5
  35. 2026CARLA-Air: Fly Drones Inside a CARLA World -- A Unified Infrastructure for Air-Ground Embodied Intelligencecited by 5
  36. 2023Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Renderingcited by 3
  37. 2024Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challengescited by 3
  38. 2024Survey on Datasets for Perception in Unstructured Outdoor Environmentscited by 3

50 most cited of 81 All 81 (JSON)