# Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition

arXiv: https://arxiv.org/abs/2305.12796 · HTML: https://mlsanity.com/paper/2305.12796/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 10 of 10)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2207.00427/) (2022)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [DSSLIC: Deep Semantic Segmentation-based Layered Image Compression](https://mlsanity.com/paper/1806.03348/) (2018)

## Built on by (most cited first, 2 of 2)

- [Goal-Oriented Semantic Communication for Wireless Image Transmission via Stable Diffusion](https://mlsanity.com/paper/2408.00428/) (2024)
- [PRISM: Distributed Inference for Foundation Models at Edge](https://mlsanity.com/paper/2507.12145/) (2025)

## Abstract (arXiv)

This paper studies the computational offloading of video action recognition in edge computing. To achieve effective semantic information extraction and compression, following semantic communication we propose a novel spatiotemporal attention-based autoencoder (STAE) architecture, including a frame attention module and a spatial attention module, to evaluate the importance of frames and pixels in each frame. Additionally, we use entropy encoding to remove statistical redundancy in the compressed data to further reduce communication overhead. At the receiver, we develop a lightweight decoder that leverages a 3D-2D CNN combined architecture to reconstruct missing information by simultaneously learning temporal and spatial information from the received data to improve accuracy. To fasten convergence, we use a step-by-step approach to train the resulting STAE-based vision transformer (ViT_STAE) models. Experimental results show that ViT_STAE can compress the video dataset HMDB51 by 104x with only 5% accuracy loss, outperforming the state-of-the-art baseline DeepISC. The proposed ViT_STAE achieves faster inference and higher accuracy than the DeepISC-based ViT model under time-varying wireless channel, which highlights the effectiveness of STAE in guaranteeing higher accuracy under time constraints.