# Semi-Supervised Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transport

arXiv: https://arxiv.org/abs/2104.08489 · HTML: https://mlsanity.com/paper/2104.08489/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [Learning with a Wasserstein Loss](https://mlsanity.com/paper/1506.05439/) (2015)
- [Fast Multi-Instance Multi-Label Learning](https://mlsanity.com/paper/1310.2049/) (2013)

## Built on by (most cited first, 28 of 28)

- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification](https://mlsanity.com/paper/2307.09066/) (2023)
- [Robust Semi-supervised Learning by Wisely Leveraging Open-set Data](https://mlsanity.com/paper/2405.06979/) (2024)
- [Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learning](https://mlsanity.com/paper/2305.02795/) (2023)
- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt](https://mlsanity.com/paper/2405.06926/) (2024)
- [Robust Semi-Supervised Learning for Self-learning Open-World Classes](https://mlsanity.com/paper/2401.07551/) (2024)
- [Robust Deep Semi-Supervised Learning: A Brief Introduction](https://mlsanity.com/paper/2202.05975/) (2022)
- [Twice Class Bias Correction for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2312.16604/) (2023)
- [The Solution for the ICCV 2023 1st Scientific Figure Captioning Challenge](https://mlsanity.com/paper/2403.17342/) (2024)
- [JobFormer: Skill-Aware Job Recommendation with Semantic-Enhanced Transformer](https://mlsanity.com/paper/2404.04313/) (2024)
- [The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge](https://mlsanity.com/paper/2406.12225/) (2024)
- [Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval](https://mlsanity.com/paper/2407.00979/) (2024)
- [Complementary Fusion of Deep Network and Tree Model for ETA Prediction](https://mlsanity.com/paper/2407.01262/) (2024)
- [Proposal Report for the 2nd SciCAP Competition 2024](https://mlsanity.com/paper/2407.01897/) (2024)
- [Visual Sensation and Perception Computational Models for Deep Learning: State of the art, Challenges and Prospects](https://mlsanity.com/paper/2109.03391/) (2021)
- [Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI](https://mlsanity.com/paper/2403.17683/) (2024)
- [The Solution for the CVPR 2023 1st foundation model challenge-Track2](https://mlsanity.com/paper/2403.17702/) (2024)
- [NJUST-KMG at TRAC-2024 Tasks 1 and 2: Offline Harm Potential Identification](https://mlsanity.com/paper/2403.19713/) (2024)
- [The Solution for the ICCV 2023 Perception Test Challenge 2023 -- Task 6 -- Grounded videoQA](https://mlsanity.com/paper/2407.01907/) (2024)
- [The Solution for Temporal Sound Localisation Task of ICCV 1st Perception Test Challenge 2023](https://mlsanity.com/paper/2407.02318/) (2024)
- [The Solution for the GAIIC2024 RGB-TIR object detection Challenge](https://mlsanity.com/paper/2407.03872/) (2024)
- [The Solution for the AIGC Inference Performance Optimization Competition](https://mlsanity.com/paper/2407.04991/) (2024)
- [The Solution for The PST-KDD-2024 OAG-Challenge](https://mlsanity.com/paper/2407.12827/) (2024)
- [Solution for Temporal Sound Localisation Task of ECCV Second Perception Test Challenge 2024](https://mlsanity.com/paper/2409.19595/) (2024)
- [The Solution for Temporal Action Localisation Task of Perception Test Challenge 2024](https://mlsanity.com/paper/2410.09088/) (2024)
- [Solution for Point Tracking Task of ECCV 2nd Perception Test Challenge 2024](https://mlsanity.com/paper/2410.16286/) (2024)
- [Solution for OOD-CV UNICORN Challenge 2024 Object Detection Assistance LLM Counting Ability Improvement](https://mlsanity.com/paper/2410.16287/) (2024)

## Abstract (arXiv)

Complex objects are usually with multiple labels, and can be represented by multiple modal representations, e.g., the complex articles contain text and image information as well as multiple annotations. Previous methods assume that the homogeneous multi-modal data are consistent, while in real applications, the raw data are disordered, e.g., the article constitutes with variable number of inconsistent text and image instances. Therefore, Multi-modal Multi-instance Multi-label (M3) learning provides a framework for handling such task and has exhibited excellent performance. However, M3 learning is facing two main challenges: 1) how to effectively utilize label correlation; 2) how to take advantage of multi-modal learning to process unlabeled instances. To solve these problems, we first propose a novel Multi-modal Multi-instance Multi-label Deep Network (M3DN), which considers M3 learning in an end-to-end multi-modal deep network and utilizes consistency principle among different modal bag-level predictions. Based on the M3DN, we learn the latent ground label metric with the optimal transport. Moreover, we introduce the extrinsic unlabeled multi-modal multi-instance data, and propose the M3DNS, which considers the instance-level auto-encoder for single modality and modified bag-level optimal transport to strengthen the consistency among modalities. Thereby M3DNS can better predict label and exploit label correlation simultaneously. Experiments on benchmark datasets and real world WKG Game-Hub dataset validate the effectiveness of the proposed methods.