mlsanityData to 7 Oct 2026

Quantized Approximately Orthogonal Recurrent Neural Networks

arXiv 2402.04012 ↗First posted 5 Feb 2024cs.NE cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices.In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 4-bits quantization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 54 of its 77 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

54 of 77 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  10. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
38 more
  1. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  2. 2019Deep Equilibrium Modelscited by 603
  3. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  4. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  7. 2023A Survey on Model Compression for Large Language Modelscited by 263
  8. 2018Sorting out Lipschitz function approximationcited by 211
  9. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  10. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  11. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  12. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  13. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  14. 2020Lipschitz Recurrent Neural Networkscited by 88
  15. 2017Dilated Recurrent Neural Networkscited by 83
  16. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  17. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  18. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 82
  19. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  20. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  21. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 62
  22. 2016Loss-aware Binarization of Deep Networkscited by 56
  23. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  24. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  25. 2018Can recurrent neural networks warp time?cited by 53
  26. 2021A comprehensive review of Binary Neural Networkcited by 44
  27. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  28. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  29. 2019Fully Quantized Transformer for Machine Translationcited by 36
  30. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  31. 2018Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representationscited by 32
  32. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  33. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 28
  34. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  35. 2017Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networkscited by 21
  36. 2021Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsitycited by 20
  37. 2017Kronecker Recurrent Unitscited by 13
  38. 2020Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translationcited by 13

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025HadamRNN: Binary and Sparse Ternary Orthogonal RNNscited by 0