mlsanityData to 7 Oct 2026

Variable-rate discrete representation learning

arXiv 2103.06089 ↗First posted 10 Mar 2021cs.LG cs.CL cs.SDcited by 24

Not on the map. Why?

Abstract (arXiv)

Semantically meaningful information content in perceptual signals is usually unevenly distributed. In speech signals for example, there are often many silences, and the speed of pronunciation can vary considerably. In this work, we propose slow autoencoders (SlowAEs) for unsupervised learning of high-level variable-rate discrete representations of sequences, and apply them to speech. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals, while still allowing for faithful signal reconstruction. We develop run-length Transformers (RLTs) for event-based representation modelling and use them to construct language models in the speech domain, which are able to generate grammatical and semantically coherent utterances and continuations.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 74 of its 115 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

74 of 115 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2019Event-based Vision: A Surveycited by 1,010
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  11. 2016End-to-end Optimized Image Compressioncited by 673
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  13. 2015Generating Sentences from a Continuous Spacecited by 607
  14. 2020Jukebox: A Generative Model for Musiccited by 535
  15. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  16. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  17. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  18. 2019PointRend: Image Segmentation as Renderingcited by 439
  19. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  20. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  21. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  22. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  23. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  24. 2017Lossy Image Compression with Compressive Autoencoderscited by 289
  25. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  26. 2021Generating Images with Sparse Representationscited by 262
  27. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  28. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  29. 2016Variational Lossy Autoencodercited by 175
  30. 2016Discrete Variational Autoencoderscited by 124
  31. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
  32. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  33. 2017Compressed Video Action Recognitioncited by 102
  34. 2015End-to-end Learning of Action Detection from Frame Glimpses in Videoscited by 101
  35. 2016Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequencescited by 98
  36. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  37. 2019Interpolation-Prediction Networks for Irregularly Sampled Time Seriescited by 89
  38. 2019Preventing Posterior Collapse with delta-VAEscited by 86

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 2; 2024: 7; 2025: 3; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  2. 2022Continuous diffusion for categorical datacited by 202
  3. 2022Generative Spoken Dialogue Language Modelingcited by 125
  4. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  5. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  6. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  7. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 44
  8. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  9. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  10. 2021Aligned Contrastive Predictive Codingcited by 15
  11. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  12. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
12 more
  1. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 7
  2. 2023Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognitioncited by 7
  3. 2024Variable Bitrate Residual Vector Quantization for Audio Codingcited by 6
  4. 2022Deep Neural Imputation: A Framework for Recovering Incomplete Brain Recordingscited by 5
  5. 2024ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Soundscited by 4
  6. 2024Exploring the Benefits of Tokenization of Discrete Acoustic Unitscited by 4
  7. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  8. 2021Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Unitscited by 1
  9. 2022Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Modelscited by 1
  10. 2024Spiking Music: Audio Compression with Event Based Auto-encoderscited by 1
  11. 2024Removing Speaker Information from Speech Representation using Variable-Length Soft Poolingcited by 1
  12. 2025Vector Quantization in the Brain: Grid-like Codes in World Modelscited by 0