# Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs

arXiv: https://arxiv.org/abs/2203.03990 · HTML: https://mlsanity.com/paper/2203.03990/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Salient Object Detection via Integrity Learning](https://mlsanity.com/paper/2101.07663/) (2021)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Learning to score the figure skating sports videos](https://mlsanity.com/paper/1802.02774/) (2018)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Kaleido-BERT: Vision-Language Pre-training on Fashion Domain](https://mlsanity.com/paper/2103.16110/) (2021)
- [Parameter Efficient Multimodal Transformers for Video Representation Learning](https://mlsanity.com/paper/2012.04124/) (2020)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [S2-MLPv2: Improved Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2108.01072/) (2021)
- [Cascaded Cross MLP-Mixer GANs for Cross-View Image Translation](https://mlsanity.com/paper/2110.10183/) (2021)
- [Action Quality Assessment using Siamese Network-Based Deep Metric Learning](https://mlsanity.com/paper/2002.12096/) (2020)
- [MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2105.14110/) (2021)
- [Mixer-TTS: non-autoregressive, fast and compact text-to-speech model conditioned on language model embeddings](https://mlsanity.com/paper/2110.03584/) (2021)
- [Estimating Blink Probability for Highlight Detection in Figure Skating Videos](https://mlsanity.com/paper/2007.01089/) (2020)

## Built on by (most cited first, 20 of 20)

- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [PHI: Bridging Domain Shift in Long-Term Action Quality Assessment via Progressive Hierarchical Instruction](https://mlsanity.com/paper/2505.19972/) (2025)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment](https://mlsanity.com/paper/2511.17397/) (2025)
- [TechCoach: Towards Technical-Point-Aware Descriptive Action Coaching](https://mlsanity.com/paper/2411.17130/) (2024)
- [FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding](https://mlsanity.com/paper/2504.19514/) (2025)
- [SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation](https://mlsanity.com/paper/2505.08665/) (2025)
- [PATS: Proficiency-Aware Temporal Sampling for Multi-View Sports Skill Assessment](https://mlsanity.com/paper/2506.04996/) (2025)
- [A Comprehensive Survey on Generative AI for Video-to-Music Generation](https://mlsanity.com/paper/2502.12489/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [ProfVLM: A lightweight video-language model for multi-view proficiency estimation](https://mlsanity.com/paper/2509.26278/) (2025)
- [Attention-Driven Multimodal Alignment for Long-term Action Quality Assessment](https://mlsanity.com/paper/2507.21945/) (2025)
- [Learning Long-Range Action Representation by Two-Stream Mamba Pyramid Network for Figure Skating Assessment](https://mlsanity.com/paper/2508.16291/) (2025)
- [Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs](https://mlsanity.com/paper/2403.17607/) (2024)
- [FineSkiing: A Fine-grained Benchmark for Skiing Action Quality Assessment](https://mlsanity.com/paper/2511.10250/) (2025)
- [CaFlow: Enhancing Long-Term Action Quality Assessment with Causal Counterfactual Flow](https://mlsanity.com/paper/2511.21653/) (2025)
- [MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment](https://mlsanity.com/paper/2605.08945/) (2026)
- [Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment](https://mlsanity.com/paper/2607.07438/) (2026)
- [MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts](https://mlsanity.com/paper/2607.19826/) (2026)
- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)

## Abstract (arXiv)

Figure skating scoring is challenging because it requires judging the technical moves of the players as well as their coordination with the background music. Most learning-based methods cannot solve it well for two reasons: 1) each move in figure skating changes quickly, hence simply applying traditional frame sampling will lose a lot of valuable information, especially in 3 to 5 minutes long videos; 2) prior methods rarely considered the critical audio-visual relationship in their models. Due to these reasons, we introduce a novel architecture, named Skating-Mixer. It extends the MLP framework into a multimodal fashion and effectively learns long-term representations through our designed memory recurrent unit (MRU). Aside from the model, we collected a high-quality audio-visual FS1000 dataset, which contains over 1000 videos on 8 types of programs with 7 different rating metrics, overtaking other datasets in both quantity and diversity. Experiments show the proposed method achieves SOTAs over all major metrics on the public Fis-V and our FS1000 dataset. In addition, we include an analysis applying our method to the recent competitions in Beijing 2022 Winter Olympic Games, proving our method has strong applicability.