# Self-attention-based non-linear basis transformations for compact latent space modelling of dynamic optical fibre transmission matrices

arXiv: https://arxiv.org/abs/2406.07775 · HTML: https://mlsanity.com/paper/2406.07775/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures](https://mlsanity.com/paper/1808.08946/) (2018)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Deep learning the high variability and randomness inside multimode fibres](https://mlsanity.com/paper/1807.09351/) (2018)
- [Transmission of natural scene images through a multimode fibre](https://mlsanity.com/paper/1904.11985/) (2019)
- [Compressively sampling the optical transmission matrix of a multimode fibre](https://mlsanity.com/paper/2007.15891/) (2020)
- [Image transmission through a flexible multimode fiber by deep learning](https://mlsanity.com/paper/2011.05144/) (2020)
- [Training Generative Adversarial Networks for Optical Property Mapping using Synthetic Image Data](https://mlsanity.com/paper/2203.07793/) (2022)
- [Characterising optical fibre transmission matrices using metasurface reflector stacks for lensless imaging without distal access](https://mlsanity.com/paper/1904.02644/) (2019)
- [Single-ended Recovery of Optical fiber Transmission Matrices using Neural Networks](https://mlsanity.com/paper/2303.17883/) (2023)

## Abstract (arXiv)

Multimode optical fibres are hair-thin strands of glass that efficiently transport light. They promise next-generation medical endoscopes that provide unprecedented sub-cellular image resolution deep inside the body. However, confining light to such fibres means that images are inherently scrambled in transit. Conventionally, this scrambling has been compensated by pre-calibrating how a specific fibre scrambles light and solving a stationary linear matrix equation that represents a physical model of the fibre. However, as the technology develops towards real-world deployment, the unscrambling process must account for dynamic changes in the matrix representing the fibre's effect on light, due to factors such as movement and temperature shifts, and non-linearities resulting from the inaccessibility of the fibre tip when inside the body. Such complex, dynamic and nonlinear behaviour is well-suited to approximation by neural networks, but most leading image reconstruction networks rely on convolutional layers, which assume strong correlations between adjacent pixels, a strong inductive bias that is inappropriate for fibre matrices which may be expressed in a range of arbitrary coordinate representations with long-range correlations. We introduce a new concept that uses self-attention layers to dynamically transform the coordinate representations of varying fibre matrices to a basis that admits compact, low-dimensional representations suitable for further processing. We demonstrate the effectiveness of this approach on diverse fibre matrix datasets. We show our models significantly improve the sparsity of fibre bases in their transformed bases with a participation ratio, p, as a measure of sparsity, of between 0.01 and 0.11. Further, we show that these transformed representations admit reconstruction of the original matrices with < 10% reconstruction error, demonstrating the invertibility.