# Early Classifying Multimodal Sequences

arXiv: https://arxiv.org/abs/2305.01151 · HTML: https://mlsanity.com/paper/2305.01151/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [N24News: A New Dataset for Multimodal News Classification](https://mlsanity.com/paper/2108.13327/) (2021)
- [OmniNet: A unified architecture for multi-modal multi-task learning](https://mlsanity.com/paper/1907.07804/) (2019)
- [Look, Listen and Learn - A Multimodal LSTM for Speaker Identification](https://mlsanity.com/paper/1602.04364/) (2016)
- [A Policy for Early Sequence Classification](https://mlsanity.com/paper/2304.03463/) (2023)

## Abstract (arXiv)

Often pieces of information are received sequentially over time. When did one collect enough such pieces to classify? Trading wait time for decision certainty leads to early classification problems that have recently gained attention as a means of adapting classification to more dynamic environments. However, so far results have been limited to unimodal sequences. In this pilot study, we expand into early classifying multimodal sequences by combining existing methods. We show our new method yields experimental AUC advantages of up to 8.7%.