# AssembleNet++: Assembling Modality Representations via Attention Connections

arXiv: https://arxiv.org/abs/2008.08072 · HTML: https://mlsanity.com/paper/2008.08072/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 28 of 28)

- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning](https://mlsanity.com/paper/2212.03229/) (2022)
- [4D-Net for Learned Multi-Modal Alignment](https://mlsanity.com/paper/2109.01066/) (2021)
- [VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living](https://mlsanity.com/paper/2105.08141/) (2021)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living](https://mlsanity.com/paper/2311.18840/) (2023)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Cross-modal Representation Learning for Zero-shot Action Recognition](https://mlsanity.com/paper/2205.01657/) (2022)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [Flow Dynamics Correction for Action Recognition](https://mlsanity.com/paper/2310.10059/) (2023)
- [Self-Supervised Video Representation Learning via Latent Time Navigation](https://mlsanity.com/paper/2305.06437/) (2023)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [Searching for Two-Stream Models in Multivariate Space for Video Recognition](https://mlsanity.com/paper/2108.12957/) (2021)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Fusion Matters: Learning Fusion in Deep Click-through Rate Prediction Models](https://mlsanity.com/paper/2411.15731/) (2024)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.03941/) (2022)
- [Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings](https://mlsanity.com/paper/2510.21424/) (2025)
- [AM Flow: Adapters for Temporal Processing in Action Recognition](https://mlsanity.com/paper/2411.02065/) (2024)
- [Auto-X3D: Ultra-Efficient Video Understanding via Finer-Grained Neural Architecture Search](https://mlsanity.com/paper/2112.04710/) (2021)
- [Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers](https://mlsanity.com/paper/2306.09331/) (2023)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)
- [Salient Temporal Encoding for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2503.14524/) (2025)
- [Language-driven Description Generation and Common Sense Reasoning for Video Action Recognition](https://mlsanity.com/paper/2506.16701/) (2025)
- [Recognition of Daily Activities through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living](https://mlsanity.com/paper/2603.04509/) (2026)

## Abstract (arXiv)

We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network. A new network component named peer-attention is introduced, which dynamically learns the attention weights using another block or input modality. Even without pre-training, our models outperform the previous work on standard public activity recognition datasets with continuous videos, establishing new state-of-the-art. We also confirm that our findings of having neural connections from the object modality and the use of peer-attention is generally applicable for different existing architectures, improving their performances. We name our model explicitly as AssembleNet++. The code will be available at: https://sites.google.com/corp/view/assemblenet/