# Exploring Long Tail Visual Relationship Recognition with Large Vocabulary

arXiv: https://arxiv.org/abs/2004.00436 · HTML: https://mlsanity.com/paper/2004.00436/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 16 of 16)

- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [Fine-Grained Scene Graph Generation with Data Transfer](https://mlsanity.com/paper/2203.11654/) (2022)
- [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognition](https://mlsanity.com/paper/2104.11934/) (2021)
- [Single-Stage Visual Relationship Learning using Conditional Queries](https://mlsanity.com/paper/2306.05689/) (2023)
- [VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage](https://mlsanity.com/paper/2510.12750/) (2025)
- [Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox](https://mlsanity.com/paper/2307.08235/) (2023)
- [Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation](https://mlsanity.com/paper/2305.18668/) (2023)
- [Towards Flexible Visual Relationship Segmentation](https://mlsanity.com/paper/2408.08305/) (2024)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [ViRED: Prediction of Visual Relations in Engineering Drawings](https://mlsanity.com/paper/2409.00909/) (2024)
- [TreeFormer: Single-view Plant Skeleton Estimation via Tree-constrained Graph Generation](https://mlsanity.com/paper/2411.16132/) (2024)
- [SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data](https://mlsanity.com/paper/2308.12910/) (2023)
- [DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition](https://mlsanity.com/paper/2401.01387/) (2024)

## Abstract (arXiv)

Several approaches have been proposed in recent literature to alleviate the long-tail problem, mainly in object classification tasks. In this paper, we make the first large-scale study concerning the task of Long-Tail Visual Relationship Recognition (LTVRR). LTVRR aims at improving the learning of structured visual relationships that come from the long-tail (e.g., "rabbit grazing on grass"). In this setup, the subject, relation, and object classes each follow a long-tail distribution. To begin our study and make a future benchmark for the community, we introduce two LTVRR-related benchmarks, dubbed VG8K-LT and GQA-LT, built upon the widely used Visual Genome and GQA datasets. We use these benchmarks to study the performance of several state-of-the-art long-tail models on the LTVRR setup. Lastly, we propose a visiolinguistic hubless (VilHub) loss and a Mixup augmentation technique adapted to LTVRR setup, dubbed as RelMix. Both VilHub and RelMix can be easily integrated on top of existing models and despite being simple, our results show that they can remarkably improve the performance, especially on tail classes. Benchmarks, code, and models have been made available at: https://github.com/Vision-CAIR/LTVRR.