# Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles

arXiv: https://arxiv.org/abs/2104.07830 · HTML: https://mlsanity.com/paper/2104.07830/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 26 of its 63 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020)
- [Control of Connected and Automated Vehicles: State of the Art and Future Challenges](https://mlsanity.com/paper/1804.03757/) (2018)

## Built on by (most cited first, 30 of 33)

- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Does Physical Adversarial Example Really Matter to Autonomous Driving? Towards System-Level Effect of Adversarial Object Evasion Attack](https://mlsanity.com/paper/2308.11894/) (2023)
- [MMFN: Multi-Modal-Fusion-Net for End-to-End Driving](https://mlsanity.com/paper/2207.00186/) (2022)
- [Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs](https://mlsanity.com/paper/2404.04869/) (2024)
- [SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads](https://mlsanity.com/paper/2312.16733/) (2023)
- [Context-Aware Streaming Perception in Dynamic Environments](https://mlsanity.com/paper/2208.07479/) (2022)
- [ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles](https://mlsanity.com/paper/2109.02529/) (2021)
- [RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics](https://mlsanity.com/paper/2308.15368/) (2023)
- [LoGoPlanner: Localization Grounded Navigation Policy with Metric-aware Visual Geometry](https://mlsanity.com/paper/2512.19629/) (2025)
- [Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems](https://mlsanity.com/paper/2311.18768/) (2023)
- [Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems](https://mlsanity.com/paper/2402.18393/) (2024)
- [MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving](https://mlsanity.com/paper/2405.07573/) (2024)
- [VALO: A Versatile Anytime Framework for LiDAR-based Object Detection Deep Neural Networks](https://mlsanity.com/paper/2409.11542/) (2024)
- [Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search](https://mlsanity.com/paper/2301.13807/) (2023)
- [Interpretable End-to-End Driving Model for Implicit Scene Understanding](https://mlsanity.com/paper/2308.01180/) (2023)
- [A Novel Multi-layer Task-centric and Data Quality Framework for Autonomous Driving](https://mlsanity.com/paper/2506.17346/) (2025)
- [Watch Out for the Safety-Threatening Actors: Proactively Mitigating Safety Hazards](https://mlsanity.com/paper/2206.00886/) (2022)
- [Reasoning-Based Software Testing](https://mlsanity.com/paper/2303.01302/) (2023)
- [DεpS: Delayed ε-Shrinking for Faster Once-For-All Training](https://mlsanity.com/paper/2407.06167/) (2024)
- [Managing Bandwidth: The Key to Cloud-Assisted Autonomous Driving](https://mlsanity.com/paper/2410.16227/) (2024)
- [Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios](https://mlsanity.com/paper/2503.23708/) (2025)
- [ADReFT: Adaptive Decision Repair for Safe Autonomous Driving via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.23960/) (2025)
- [A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems](https://mlsanity.com/paper/2509.22379/) (2025)
- [SEO: Safety-Aware Energy Optimization Framework for Multi-Sensor Neural Controllers at the Edge](https://mlsanity.com/paper/2302.12493/) (2023)
- [A Framework for Benchmarking Real-Time Embedded Object Detection](https://mlsanity.com/paper/2304.11580/) (2023)
- [Tuning the feedback controller gains is a simple way to improve autonomous driving performance](https://mlsanity.com/paper/2402.05064/) (2024)
- [STCLocker: Deadlock Avoidance Testing for Autonomous Driving Systems](https://mlsanity.com/paper/2506.23995/) (2025)
- [Multi-Resolution End-to-End Deep Neural Network for Optimizing Latency-Accuracy Tradeoff in Autonomous Driving](https://mlsanity.com/paper/2605.29138/) (2026)
- [CADET: A Modular Platform for Evaluating Distributed Cooperative Autonomy in Connected Autonomous Vehicles](https://mlsanity.com/paper/2606.04072/) (2026)

## Abstract (arXiv)

We present Pylot, a platform for autonomous vehicle (AV) research and development, built with the goal to allow researchers to study the effects of the latency and accuracy of their models and algorithms on the end-to-end driving behavior of an AV. This is achieved through a modular structure enabled by our high-performance dataflow system that represents AV software pipeline components (object detectors, motion planners, etc.) as a dataflow graph of operators which communicate on data streams using timestamped messages. Pylot readily interfaces with popular AV simulators like CARLA, and is easily deployable to real-world vehicles with minimal code changes. To reduce the burden of developing an entire pipeline for evaluating a single component, Pylot provides several state-of-the-art reference implementations for the various components of an AV pipeline. Using these reference implementations, a Pylot-based AV pipeline is able to drive a real vehicle, and attains a high score on the CARLA Autonomous Driving Challenge. We also present several case studies enabled by Pylot, including evidence of a need for context-dependent components, and per-component time allocation. Pylot is open source, with the code available at https://github.com/erdos-project/pylot.