# Relation-Aware Distribution Representation Network for Person Clustering with Multiple Modalities

arXiv: https://arxiv.org/abs/2308.00588 · HTML: https://mlsanity.com/paper/2308.00588/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 88 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification](https://mlsanity.com/paper/2001.01526/) (2020)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [DPGN: Distribution Propagation Graph Network for Few-shot Learning](https://mlsanity.com/paper/2003.14247/) (2020)
- [Linkage Based Face Clustering via Graph Convolution Network](https://mlsanity.com/paper/1903.11306/) (2019)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Learning to Cluster Faces on an Affinity Graph](https://mlsanity.com/paper/1904.02749/) (2019)
- [Learning to Cluster Faces via Confidence and Connectivity Estimation](https://mlsanity.com/paper/2004.00445/) (2020)
- [Modality Shifting Attention Network for Multi-modal Video Question Answering](https://mlsanity.com/paper/2007.02036/) (2020)
- [Clustering Millions of Faces by Identity](https://mlsanity.com/paper/1604.00989/) (2016)
- [Label Distribution Learning Forests](https://mlsanity.com/paper/1702.06086/) (2017)
- [Principled Multilayer Network Embedding](https://mlsanity.com/paper/1709.03551/) (2017)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [Structure-Aware Face Clustering on a Large-Scale Graph with bf107 Nodes](https://mlsanity.com/paper/2103.13225/) (2021)
- [Person Search in Videos with One Portrait Through Visual and Temporal Links](https://mlsanity.com/paper/1807.10510/) (2018)
- [A Proximity-Aware Hierarchical Clustering of Faces](https://mlsanity.com/paper/1703.04835/) (2017)
- [On the Selection of Anchors and Targets for Video Hyperlinking](https://mlsanity.com/paper/1804.05286/) (2018)
- [Learning to Cluster Faces via Transformer](https://mlsanity.com/paper/2104.11502/) (2021)

## Built on by (most cited first, 2 of 2)

- [Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer](https://mlsanity.com/paper/2407.18554/) (2024)
- [Exploring Genre and Success Classification through Song Lyrics using DistilBERT: A Fun NLP Venture](https://mlsanity.com/paper/2407.21068/) (2024)

## Abstract (arXiv)

Person clustering with multi-modal clues, including faces, bodies, and voices, is critical for various tasks, such as movie parsing and identity-based movie editing. Related methods such as multi-view clustering mainly project multi-modal features into a joint feature space. However, multi-modal clue features are usually rather weakly correlated due to the semantic gap from the modality-specific uniqueness. As a result, these methods are not suitable for person clustering. In this paper, we propose a Relation-Aware Distribution representation Network (RAD-Net) to generate a distribution representation for multi-modal clues. The distribution representation of a clue is a vector consisting of the relation between this clue and all other clues from all modalities, thus being modality agnostic and good for person clustering. Accordingly, we introduce a graph-based method to construct distribution representation and employ a cyclic update policy to refine distribution representation progressively. Our method achieves substantial improvements of +6% and +8.2% in F-score on the Video Person-Clustering Dataset (VPCD) and VoxCeleb2 multi-view clustering dataset, respectively. Codes will be released publicly upon acceptance.