Interpreting A Pre-trained Model Is A Key For Model Architecture Optimization: A Case Study On Wav2Vec 2.0
Not on the map. Why?
Abstract (arXiv)
A deep Transformer model with good evaluation score does not mean each subnetwork (a.k.a transformer block) learns reasonable representation. Diagnosing abnormal representation and avoiding it can contribute to achieving a better evaluation score. We propose an innovative perspective for analyzing attention patterns: summarize block-level patterns and assume abnormal patterns contribute negative influence. We leverage Wav2Vec 2.0 as a research target and analyze a pre-trained model's pattern. All experiments leverage Librispeech-100-clean as training data. Through avoiding diagnosed abnormal ones, our custom Wav2Vec 2.0 outperforms the original version about 4.8% absolute word error rate (WER) on test-clean with viterbi decoding. Our version is still 0.9% better when decoding with a 4-gram language model. Moreover, we identify that avoiding abnormal patterns is the main contributor for performance boosting.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 7 Apr 2021.
Builds on
19 of 26 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
7 more
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2018Image Transformercited by 772
- 2017Dynamic Routing Between Capsulescited by 689
- 2018Video Action Transformer Networkcited by 316
- 2019Revealing the Dark Secrets of BERTcited by 301
- 2015Visualizing and Understanding Recurrent Networkscited by 160
- 2018Did the Model Understand the Question?cited by 60
Built on by
Not cited yet.