# Sigma Flows for Image and Data Labeling and Learning Structured Prediction

arXiv: https://arxiv.org/abs/2408.15946 · HTML: https://mlsanity.com/paper/2408.15946/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 91 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Beltrami Flow and Neural Diffusion on Graphs](https://mlsanity.com/paper/2110.09443/) (2021)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [PDE-based Group Equivariant Convolutional Neural Networks](https://mlsanity.com/paper/2001.09046/) (2020)
- [From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond](https://mlsanity.com/paper/2310.10121/) (2023)
- [Image Labeling by Assignment](https://mlsanity.com/paper/1603.05285/) (2016)
- [Assignment Flows for Data Labeling on Graphs: Convergence and Stability](https://mlsanity.com/paper/2002.11571/) (2020)
- [A Geometric Embedding Approach to Multiple Games and Multiple Populations](https://mlsanity.com/paper/2401.05918/) (2024)
- [Generative Assignment Flows for Representing and Learning Joint Distributions of Discrete Data](https://mlsanity.com/paper/2406.04527/) (2024)
- [Highly corrupted image inpainting through hypoelliptic diffusion](https://mlsanity.com/paper/1502.07331/) (2015)
- [Collaborative Total Variation: A General Framework for Vectorial TV Models](https://mlsanity.com/paper/1508.01308/) (2015)
- [Continuous-Domain Assignment Flows](https://mlsanity.com/paper/1910.07287/) (2019)
- [Learning Adaptive Regularization for Image Labeling Using Geometric Assignment](https://mlsanity.com/paper/1910.09976/) (2019)
- [Unsupervised Assignment Flow: Label Learning on Feature Manifolds by Spatially Regularized Geometric Assignment](https://mlsanity.com/paper/1904.10863/) (2019)
- [Self-Certifying Classification by Linearized Deep Assignment](https://mlsanity.com/paper/2201.11162/) (2022)
- [Quantum State Assignment Flows](https://mlsanity.com/paper/2307.00075/) (2023)

## Built on by (most cited first, 1 of 1)

- [Riemannian Patch Assignment Gradient Flows](https://mlsanity.com/paper/2504.13024/) (2025)

## Abstract (arXiv)

This paper introduces the sigma flow model for the prediction of structured labelings of data observed on Riemannian manifolds, including Euclidean image domains as special case. The approach combines the Laplace-Beltrami framework for image denoising and enhancement, introduced by Sochen, Kimmel and Malladi about 25 years ago, and the assignment flow approach introduced and studied by the authors. The sigma flow arises as Riemannian gradient flow of generalized harmonic energies and thus is governed by a nonlinear geometric PDE which determines a harmonic map from a closed Riemannian domain manifold to a statistical manifold, equipped with the Fisher-Rao metric from information geometry. A specific ingredient of the sigma flow is the mutual dependency of the Riemannian metric of the domain manifold on the evolving state. This makes the approach amenable to machine learning in a specific way, by realizing this dependency through a mapping with compact time-variant parametrization that can be learned from data. Proof of concept experiments demonstrate the expressivity of the sigma flow model and prediction performance. Structural similarities to transformer network architectures and networks generated by the geometric integration of sigma flows are pointed out, which highlights the connection to deep learning and, conversely, may stimulate the use of geometric design principles for structured prediction in other areas of scientific machine learning.