# Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition

arXiv: https://arxiv.org/abs/2609.29443 · HTML: https://mlsanity.com/paper/2609.29443/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Task-adaptive Neural Process for User Cold-Start Recommendation](https://mlsanity.com/paper/2103.06137/) (2021)
- [SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision](https://mlsanity.com/paper/2303.17200/) (2023)
- [SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization](https://mlsanity.com/paper/2406.12233/) (2024)
- [Towards Pose-invariant Lip-Reading](https://mlsanity.com/paper/1911.06095/) (2019)
- [Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping](https://mlsanity.com/paper/2308.06112/) (2023)
- [SparseVSR: Lightweight and Noise Robust Visual Speech Recognition](https://mlsanity.com/paper/2307.04552/) (2023)
- [Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction](https://mlsanity.com/paper/2507.18863/) (2025)
- [LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data](https://mlsanity.com/paper/2312.09727/) (2023)
- [Expression-preserving face frontalization improves visually assisted speech processing](https://mlsanity.com/paper/2204.02810/) (2022)
- [LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition](https://mlsanity.com/paper/2501.04204/) (2025)
- [GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition](https://mlsanity.com/paper/2509.16031/) (2025)

## Abstract (arXiv)

Head-pose variation introduces substantial appearance transformations in visual speech recognition (VSR), making pose-aware feature modulation desirable. However, performance degradation and unwanted feature interactions may result from using numerous Feature-wise Linear Modulation (FiLM) circuits with fixed modulation intensity. We propose a Pose Adaptive Dynamic FiLM framework with a Dynamic Residual FiLM (DR-FiLM) modulator that predicts input-dependent weights to adaptively control the strength of pose-conditioned modulation. Experiments on LRS2 and LRS3 demonstrate that unweighted multi-pathway modulation substantially degrades phoneme recognition, increasing PER to 20.33% and 29.42%, respectively, compared with 16.20% and 20.96% for the single ResFiLM configuration. In contrast, the proposed DR-FiLM with dynamic Deep-Res weighting reduces PER to 15.74% on LRS2 and 23.91% on LRS3, substantially mitigating the adverse effects of unweighted modulation. The analysis of the learned weights further reveals a consistent tendency to assign greater weight to the deeper FiLM pathway as head-pose variation increases. These results show that merging pose-conditioned FiLM circuits is more efficient when the modulation strength is dynamically controlled.