# Describing Multimedia Content using Attention-based Encoder--Decoder Networks

arXiv: https://arxiv.org/abs/1507.01053 · HTML: https://mlsanity.com/paper/1507.01053/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2015.

## Built on by (most cited first, 30 of 32)

- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns](https://mlsanity.com/paper/2310.01749/) (2023)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [Beyond the C: Retargetable Decompilation using Neural Machine Translation](https://mlsanity.com/paper/2212.08950/) (2022)
- [AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation](https://mlsanity.com/paper/2210.02578/) (2022)
- [An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation](https://mlsanity.com/paper/1908.10835/) (2019)
- [AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation](https://mlsanity.com/paper/2110.11474/) (2021)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Exploitation-Guided Exploration for Semantic Embodied Navigation](https://mlsanity.com/paper/2311.03357/) (2023)
- [Comateformer: Combined Attention Transformer for Semantic Sentence Matching](https://mlsanity.com/paper/2412.07220/) (2024)
- [Recommender systems based on graph embedding techniques: A comprehensive review](https://mlsanity.com/paper/2109.09587/) (2021)
- [Set-to-Sequence Methods in Machine Learning: a Review](https://mlsanity.com/paper/2103.09656/) (2021)
- [Memory-Based Semantic Parsing](https://mlsanity.com/paper/2110.07358/) (2021)
- [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022)
- [Composable Generative Models](https://mlsanity.com/paper/2102.09249/) (2021)
- [LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching](https://mlsanity.com/paper/2108.02915/) (2021)
- [The Role of the Input in Natural Language Video Description](https://mlsanity.com/paper/2102.05067/) (2021)
- [Audio Captioning with Composition of Acoustic and Semantic Information](https://mlsanity.com/paper/2105.06355/) (2021)
- [Cross Modification Attention Based Deliberation Model for Image Captioning](https://mlsanity.com/paper/2109.08411/) (2021)
- [Dynamic and Context-Dependent Stock Price Prediction Using Attention Modules and News Sentiment](https://mlsanity.com/paper/2205.01639/) (2022)
- [Rethinking Trajectory Prediction via "Team Game"](https://mlsanity.com/paper/2210.08793/) (2022)
- [Neural Twins Talk & Alternative Calculations](https://mlsanity.com/paper/2108.02807/) (2021)
- [Learning to Minimize the Remainder in Supervised Learning](https://mlsanity.com/paper/2201.09193/) (2022)
- [Towards Micro-video Thumbnail Selection via a Multi-label Visual-semantic Embedding Model](https://mlsanity.com/paper/2202.02930/) (2022)
- [Cross-Modal Reasoning with Event Correlation for Video Question Answering](https://mlsanity.com/paper/2312.12721/) (2023)
- [Compressed Image Captioning using CNN-based Encoder-Decoder Framework](https://mlsanity.com/paper/2404.18062/) (2024)

## Abstract (arXiv)

Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output problems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input. We focus in this paper on the case where the input also has a rich structure and the input and output structures are somehow related. We describe systems that learn to attend to different places in the input, for each element of the output, for a variety of tasks: machine translation, image caption generation, video clip description and speech recognition. All these systems are based on a shared set of building blocks: gated recurrent neural networks and convolutional neural networks, along with trained attention mechanisms. We report on experimental results with these systems, showing impressively good performance and the advantage of the attention mechanism.