mlsanityData to 14 Oct 2026

Does computer vision matter for action?

arXiv 1905.12887 ↗First posted 30 May 2019cs.CV cs.AI cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Computer vision produces representations of scene content. Much computer vision research is predicated on the assumption that these intermediate representations are useful for action. Recent work at the intersection of machine learning and robotics calls this assumption into question by training sensorimotor systems directly for the task at hand, from pixels to actions, with no explicit intermediate representations. Thus the central question of our work: Does computer vision matter for action? We probe this question and its offshoots via immersive simulation, which allows us to conduct controlled reproducible experiments at scale. We instrument immersive three-dimensional environments to simulate challenges such as urban driving, off-road trail traversal, and battle. Our main finding is that computer vision does matter. Models equipped with intermediate representations train faster, achieve higher task performance, and generalize better to previously unseen environments. A video that summarizes the work and illustrates the results can be found at https://youtu.be/4MfWa2yZ0Jc

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 16; 2022: 13; 2023: 12; 2024: 11; 2025: 7; 2026: 3.

64 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  3. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 382
  4. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  5. 2022Masked Visual Pre-training for Motor Controlcited by 248
  6. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 173
  7. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  8. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  9. 2021Learning High-Speed Flight in the Wildcited by 128
  10. 2022P3Depth: Monocular Depth Estimation with a Piecewise Planarity Priorcited by 107
  11. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  12. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
38 more
  1. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  2. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  3. 2023iDisc: Internal Discretization for Monocular Depth Estimationcited by 73
  4. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 67
  5. 2025UniK3D: Universal Camera Monocular 3D Estimationcited by 62
  6. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  7. 2021Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networkscited by 52
  8. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 51
  9. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  10. 2024MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertaintycited by 32
  11. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 24
  12. 2024Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RLcited by 23
  13. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23
  14. 2021Deep Multi-Task Learning for Joint Localization, Perception, and Predictioncited by 22
  15. 2021Learning by Watchingcited by 22
  16. 2021ACDC: The Adverse Conditions Dataset with Correspondences for Robust Semantic Driving Scene Perceptioncited by 21
  17. 2021Representation Learning for Event-based Visuomotor Policiescited by 20
  18. 2023Coaching a Teachable Studentcited by 19
  19. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 18
  20. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 15
  21. 2021Reimagining an autonomous vehiclecited by 11
  22. 20253D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detectioncited by 11
  23. 2023PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenescited by 10
  24. 2021DROID: Driver-centric Risk Object Identificationcited by 9
  25. 2020Driving Through Ghosts: Behavioral Cloning with False Positivescited by 8
  26. 2020Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectorycited by 8
  27. 2021Learning Continuous Environment Fields via Implicit Functionscited by 8
  28. 2024CarFormer: Self-Driving with Learned Object-Centric Representationscited by 8
  29. 2022Practical Imitation Learning in the Real World via Task Consistency Losscited by 7
  30. 2022LiDAR-as-Camera for End-to-End Drivingcited by 7
  31. 2024When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?cited by 7
  32. 2021Navigation-Oriented Scene Understanding for Robotic Autonomy: Learning to Segment Driveability in Egocentric Imagescited by 6
  33. 2022GraphMapper: Efficient Visual Navigation by Scene Graph Generationcited by 6
  34. 2021Object-centric Video Prediction without Annotationcited by 5
  35. 2025DGFusion: Depth-Guided Sensor Fusion for Robust Semantic Perceptioncited by 4
  36. 2026SPEAR: A Simulator for Photorealistic Embodied AI Researchcited by 4
  37. 2025Video Depth Propagationcited by 3
  38. 2021Pre-training of Deep RL Agents for Improved Learning under Domain Randomizationcited by 2

50 most cited of 64 All 64 (JSON)