# X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics arXiv: https://arxiv.org/abs/2108.08217 ยท HTML: https://mlsanity.com/paper/2108.08217/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 30 references that are in the map. - First posted on arXiv 18 Aug 2021. ## Builds on (most cited first, 27 of 27) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015) - [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016) - [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019) - [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019) - [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019) - [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018) - [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018) - [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020) - [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014) - [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015) - [Boosting Image Captioning with Attributes](https://mlsanity.com/paper/1611.01646/) (2016) - [Jointly Localizing and Describing Events for Dense Video Captioning](https://mlsanity.com/paper/1804.08274/) (2018) - [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015) - [Convolutional Image Captioning](https://mlsanity.com/paper/1711.09151/) (2017) - [Video Captioning with Transferred Semantic Attributes](https://mlsanity.com/paper/1611.07675/) (2016) - [Hierarchy Parsing for Image Captioning](https://mlsanity.com/paper/1909.03918/) (2019) - [Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training](https://mlsanity.com/paper/2007.02375/) (2020) - [Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects](https://mlsanity.com/paper/1708.05271/) (2017) - [Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network](https://mlsanity.com/paper/2101.11562/) (2021) - [Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning](https://mlsanity.com/paper/1905.01077/) (2019) ## Built on by (most cited first, 10 of 10) - [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023) - [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023) - [LAVIS: A Library for Language-Vision Intelligence](https://mlsanity.com/paper/2209.09019/) (2022) - [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022) - [Comprehending and Ordering Semantics for Image Captioning](https://mlsanity.com/paper/2206.06930/) (2022) - [Sports Video Analysis on Large-Scale Data](https://mlsanity.com/paper/2208.04897/) (2022) - [Boosting Video-Text Retrieval with Explicit High-Level Semantics](https://mlsanity.com/paper/2208.04215/) (2022) - [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022) - [Long-term Leap Attention, Short-term Periodic Shift for Video Classification](https://mlsanity.com/paper/2207.05526/) (2022) - [Symmetric Network with Spatial Relationship Modeling for Natural Language-based Vehicle Retrieval](https://mlsanity.com/paper/2206.10879/) (2022) ## Abstract (arXiv) With the rise and development of deep learning over the past decade, there has been a steady momentum of innovation and breakthroughs that convincingly push the state-of-the-art of cross-modal analytics between vision and language in multimedia field. Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion. In this work, we propose X-modaler -- a versatile and high-performance codebase that encapsulates the state-of-the-art cross-modal analytics into several general-purpose stages (e.g., pre-processing, encoder, cross-modal interaction, decoder, and decode strategy). Each stage is empowered with the functionality that covers a series of modules widely adopted in state-of-the-arts and allows seamless switching in between. This way naturally enables a flexible implementation of state-of-the-art algorithms for image captioning, video captioning, and vision-language pre-training, aiming to facilitate the rapid development of research community. Meanwhile, since the effective modular designs in several stages (e.g., cross-modal interaction) are shared across different vision-language tasks, X-modaler can be simply extended to power startup prototypes for other tasks in cross-modal analytics, including visual question answering, visual commonsense reasoning, and cross-modal retrieval. X-modaler is an Apache-licensed codebase, and its source codes, sample projects and pre-trained models are available on-line: https://github.com/YehLi/xmodaler.