# Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

arXiv: https://arxiv.org/abs/2208.12866 · HTML: https://mlsanity.com/paper/2208.12866/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 139 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 30 of 53)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Methods for Pruning Deep Neural Networks](https://mlsanity.com/paper/2011.00241/) (2020)
- [Towards Efficient Post-training Quantization of Pre-trained Language Models](https://mlsanity.com/paper/2109.15082/) (2021)
- [Training Quantized Neural Networks with a Full-precision Auxiliary Module](https://mlsanity.com/paper/1903.11236/) (2019)
- [Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions](https://mlsanity.com/paper/1806.09228/) (2018)
- [Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference](https://mlsanity.com/paper/2102.11289/) (2021)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)

## Built on by (most cited first, 7 of 7)

- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Knowledge Distillation Applied to Optical Channel Equalization: Solving the Parallelization Problem of Recurrent Connection](https://mlsanity.com/paper/2212.04569/) (2022)
- [Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays](https://mlsanity.com/paper/2212.04703/) (2022)
- [Application of Transformers for Nonlinear Channel Compensation in Optical Systems](https://mlsanity.com/paper/2304.13119/) (2023)
- [Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks](https://mlsanity.com/paper/2410.11203/) (2024)
- [Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments](https://mlsanity.com/paper/2310.05950/) (2023)
- [Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation](https://mlsanity.com/paper/2408.02685/) (2024)

## Abstract (arXiv)

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in high-speed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.