# Learning long range dependencies through time reversal symmetry breaking

arXiv: https://arxiv.org/abs/2506.05259 · HTML: https://mlsanity.com/paper/2506.05259/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [The Elements of Differentiable Programming](https://mlsanity.com/paper/2403.14606/) (2024)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Oscillatory State-Space Models](https://mlsanity.com/paper/2410.03943/) (2024)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference](https://mlsanity.com/paper/2402.18512/) (2024)
- [Self-learning Machines based on Hamiltonian Echo Backpropagation](https://mlsanity.com/paper/2103.04992/) (2021)

## Built on by (most cited first, 7 of 7)

- [Local Learning Rules for Out-of-Equilibrium Physical Generative Models](https://mlsanity.com/paper/2506.19136/) (2025)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [ASecond-Order SpikingSSM for Wearables](https://mlsanity.com/paper/2510.14386/) (2025)
- [Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrence](https://mlsanity.com/paper/2605.16048/) (2026)
- [Equilibrium Propagation and Hamiltonian Inference in the Diffusive Fitzhugh-Nagumo Model](https://mlsanity.com/paper/2605.21568/) (2026)
- [Unifying Physical Backpropagation](https://mlsanity.com/paper/2608.11585/) (2026)
- [Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks](https://mlsanity.com/paper/2609.04134/) (2026)

## Abstract (arXiv)

Deep State Space Models (SSMs) reignite physics-grounded compute paradigms, as RNNs could natively be embodied into dynamical systems. This calls for dedicated learning algorithms obeying to core physical principles, with efficient techniques to simulate these systems and guide their design. We propose Recurrent Hamiltonian Echo Learning (RHEL), an algorithm which provably computes loss gradients as finite differences of physical trajectories of non-dissipative, Hamiltonian systems. In ML terms, RHEL only requires three "forward passes" irrespective of model size, without explicit Jacobian computation, nor incurring any variance in the gradient estimation. Motivated by the physical realization of our algorithm, we first introduce RHEL in continuous time and demonstrate its formal equivalence with the continuous adjoint state method. To facilitate the simulation of Hamiltonian systems trained by RHEL, we propose a discrete-time version of RHEL which is equivalent to Backpropagation Through Time (BPTT) when applied to a class of recurrent modules which we call Hamiltonian Recurrent Units (HRUs). This setting allows us to demonstrate the scalability of RHEL by generalizing these results to hierarchies of HRUs, which we call Hamiltonian SSMs (HSSMs). We apply RHEL to train HSSMs with linear and nonlinear dynamics on a variety of time-series tasks ranging from mid-range to long-range classification and regression with sequence length reaching ~ 50k. We show that RHEL consistently matches the performance of BPTT across all models and tasks. This work opens new doors for the design of scalable, energy-efficient physical systems endowed with self-learning capabilities for sequence modelling.