Discriminative Semantic Transitive Consistency for Cross-Modal Learning
Not on the map. Why?
Abstract (arXiv)
Cross-modal retrieval is generally performed by projecting and aligning the data from two different modalities onto a shared representation space. This shared space often also acts as a bridge for translating the modalities. We address the problem of learning such representation space by proposing and exploiting the property of Discriminative Semantic Transitive Consistency -- ensuring that the data points are correctly classified even after being transferred to the other modality. Along with semantic transitive consistency, we also enforce the traditional distance minimizing constraint which makes the projections of the corresponding data points from both the modalities to come closer in the representation space. We analyze and compare the contribution of both the loss terms and their interaction, for the task. In addition, we incorporate semantic cycle-consistency for each of the modality. We empirically demonstrate better performance owing to the different components with clear ablation studies. We also provide qualitative results to support the proposals.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 25 Mar 2021.
Builds on
42 of 64 references.
References · 42
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
- 2017Look, Listen and Learncited by 452
- 2018Video-to-Video Synthesiscited by 388
- 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
- 2018Everybody Dance Nowcited by 313
- 2018The Sound of Pixelscited by 258
- 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
- 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
- 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
30 more
- 2017Real-Time User-Guided Image Colorization with Learned Deep Priorscited by 123
- 2020Multi-Modal Domain Adaptation for Fine-Grained Action Recognitioncited by 115
- 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
- 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
- 2018Multi-modal Cycle-consistent Generalized Zero-Shot Learningcited by 87
- 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
- 2018A Zero-Shot Framework for Sketch-based Image Retrievalcited by 70
- 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
- 2019Canonical Surface Mapping via Geometric Cycle Consistencycited by 62
- 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
- 2019Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrievalcited by 57
- 2016Deep Cross-Modal Hashingcited by 54
- 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
- 2018Emotion Recognition in Speech using Cross-Modal Transfer in the Wildcited by 48
- 2017CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generationcited by 31
- 2019Dance Dance Generation: Motion Transfer for Internet Videoscited by 28
- 2019Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videoscited by 27
- 2020Speech2Action: Cross-modal Supervision for Action Recognitioncited by 26
- 2021Beyond Image to Depth: Improving Depth Prediction using Echoescited by 24
- 2018Disjoint Mapping Network for Cross-modal Matching of Voices and Facescited by 22
- 2018Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusioncited by 21
- 2020AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddingscited by 21
- 2017Knowledge Transfer from Weakly Labeled Audio using Convolutional Neural Network for Sound Events and Scenescited by 20
- 2016Cross-Modal Scene Networkscited by 16
- 2018Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrievalcited by 11
- 2017CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Networkcited by 10
- 2018MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrievalcited by 8
- 2019Generative Model for Zero-Shot Sketch-Based Image Retrievalcited by 7
- 2019Coupled CycleGAN: Unsupervised Hashing Network for Cross-Modal Retrievalcited by 5
- 2017Simple to Complex Cross-modal Learning to Rankcited by 3
Built on by
Citing papers per year: 2021: 1.
1 paper.