mlsanityData to 7 Oct 2026

Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems

arXiv 2104.04000 ↗First posted 8 Apr 2021cs.AI cs.LG cs.ROcited by 4

Not on the map. Why?

Abstract (arXiv)

Optimizing the quality of result (QoR) and the quality of service (QoS) of AI-empowered autonomous systems simultaneously is very challenging. First, there are multiple input sources, e.g., multi-modal data from different sensors, requiring diverse data preprocessing, sensor fusion, and feature aggregation. Second, there are multiple tasks that require various AI models to run simultaneously, e.g., perception, localization, and control. Third, the computing and control system is heterogeneous, composed of hardware components with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated accelerators. Therefore, autonomous systems essentially require multi-modal multi-task (MMMT) learning which must be aware of hardware performance and implementation strategies. While MMMT learning has been attracting intensive research interests, its applications in autonomous systems are still underexplored. In this paper, we first discuss the opportunities of applying MMMT techniques in autonomous systems and then discuss the unique challenges that must be solved. In addition, we discuss the necessity and opportunities of MMMT model and hardware co-design, which is critical for autonomous systems especially with power/resource-limited or heterogeneous platforms. We formulate the MMMT model and heterogeneous hardware implementation co-design as a differentiable optimization problem, with the objective of improving the solution quality and reducing the overall power consumption and critical path latency. We advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

37 of 62 references.

References · 37

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2017A Survey on Multi-Task Learningcited by 838
  7. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  9. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  10. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  11. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  12. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
25 more
  1. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  2. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  3. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  4. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  5. 2020CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detectioncited by 159
  6. 2014A* Samplingcited by 150
  7. 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
  8. 2016Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classificationcited by 93
  9. 2017One Model To Learn Them Allcited by 86
  10. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  11. 2020A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detectioncited by 70
  12. 2018VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometrycited by 61
  13. 2020MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learningcited by 50
  14. 2020Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understandingcited by 42
  15. 2020EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutionscited by 35
  16. 2020Deep Multimodal Neural Architecture Searchcited by 34
  17. 2018End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptioncited by 27
  18. 2018Dynamic Multi-Level Multi-Task Learning for Sentence Simplificationcited by 25
  19. 2019OmniNet: A unified architecture for multi-modal multi-task learningcited by 25
  20. 2018Evolutionary Architecture Search For Deep Multitask Networkscited by 23
  21. 2018Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networkscited by 19
  22. 2019NAIS: Neural Architecture and Implementation Search and its Applications in Autonomous Drivingcited by 9
  23. 2020FusionLane: Multi-Sensor Fusion for Lane Marking Semantic Segmentation Using Deep Neural Networkscited by 5
  24. 2020Context-Aware Multi-Task Learning for Traffic Scene Recognition in Autonomous Vehiclescited by 3
  25. 2019Hierarchical Multi-task Deep Neural Network Architecture for End-to-End Drivingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1; 2025: 1.

4 papers.

  1. 2022H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awarenesscited by 6
  2. 2024Learning 3D Perception from Others' Predictionscited by 4
  3. 2025A Comprehensive Review of Techniques, Algorithms, Advancements, Challenges, and Clinical Applications of Multi-modal Medical Image Fusion for Improved Diagnosiscited by 2
  4. 2022Compilation and Optimizations for Efficient Machine Learning on Embedded Systemscited by 0