# Multi-Perspective LSTM for Joint Visual Representation Learning

arXiv: https://arxiv.org/abs/2105.02802 · HTML: https://mlsanity.com/paper/2105.02802/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [Reasoning about Entailment with Neural Attention](https://mlsanity.com/paper/1509.06664/) (2015)
- [Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates](https://mlsanity.com/paper/1706.08276/) (2017)
- [Speech Emotion Recognition with Dual-Sequence LSTM Architecture](https://mlsanity.com/paper/1910.08874/) (2019)
- [Lipper: Synthesizing Thy Speech using Multi-View Lipreading](https://mlsanity.com/paper/1907.01367/) (2019)
- [Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition](https://mlsanity.com/paper/1704.01194/) (2017)
- [CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing](https://mlsanity.com/paper/2101.03503/) (2021)
- [End-to-End Multi-View Lipreading](https://mlsanity.com/paper/1709.00443/) (2017)
- [A Double-Deep Spatio-Angular Learning Framework for Light Field based Face Recognition](https://mlsanity.com/paper/1805.10078/) (2018)
- [Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System](https://mlsanity.com/paper/1710.07161/) (2017)
- ["Notic My Speech" -- Blending Speech Patterns With Multimedia](https://mlsanity.com/paper/2006.08599/) (2020)

## Built on by (most cited first, 3 of 3)

- [Self-supervised Contrastive Learning of Multi-view Facial Expressions](https://mlsanity.com/paper/2108.06723/) (2021)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [Contrastive Learning of View-Invariant Representations for Facial Expressions Recognition](https://mlsanity.com/paper/2311.06852/) (2023)

## Abstract (arXiv)

We present a novel LSTM cell architecture capable of learning both intra- and inter-perspective relationships available in visual sequences captured from multiple perspectives. Our architecture adopts a novel recurrent joint learning strategy that uses additional gates and memories at the cell level. We demonstrate that by using the proposed cell to create a network, more effective and richer visual representations are learned for recognition tasks. We validate the performance of our proposed architecture in the context of two multi-perspective visual recognition tasks namely lip reading and face recognition. Three relevant datasets are considered and the results are compared against fusion strategies, other existing multi-input LSTM architectures, and alternative recognition solutions. The experiments show the superior performance of our solution over the considered benchmarks, both in terms of recognition accuracy and complexity. We make our code publicly available at https://github.com/arsm/MPLSTM.