# Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving

arXiv: https://arxiv.org/abs/2002.03629 · HTML: https://mlsanity.com/paper/2002.03629/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 30 of 52)

- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Concrete Score Matching: Generalized Score Matching for Discrete Data](https://mlsanity.com/paper/2211.00802/) (2022)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Parallel Sampling of Diffusion Models](https://mlsanity.com/paper/2305.16317/) (2023)
- [SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL](https://mlsanity.com/paper/2504.11455/) (2025)
- [CLLMs: Consistency Large Language Models](https://mlsanity.com/paper/2403.00835/) (2024)
- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://mlsanity.com/paper/2410.01699/) (2024)
- [Parallelized Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15119/) (2024)
- [Parallelizing non-linear sequential models over the sequence length](https://mlsanity.com/paper/2309.12252/) (2023)
- [Towards Scalable and Stable Parallelization of Nonlinear RNNs](https://mlsanity.com/paper/2407.19115/) (2024)
- [Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling](https://mlsanity.com/paper/2507.17801/) (2025)
- [Accelerating Parallel Sampling of Diffusion Models](https://mlsanity.com/paper/2402.09970/) (2024)
- [DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling](https://mlsanity.com/paper/2311.17082/) (2023)
- [Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching](https://mlsanity.com/paper/2412.17153/) (2024)
- [Grouped Speculative Decoding for Autoregressive Image Generation](https://mlsanity.com/paper/2508.07747/) (2025)
- [ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models](https://mlsanity.com/paper/2510.21450/) (2025)
- [Fast and Accurate Causal Parallel Decoding using Jacobi Forcing](https://mlsanity.com/paper/2512.14681/) (2025)
- [DeepPCR: Parallelizing Sequential Operations in Neural Networks](https://mlsanity.com/paper/2309.16318/) (2023)
- [Inference Optimization of Foundation Models on AI Accelerators](https://mlsanity.com/paper/2407.09111/) (2024)
- [STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows](https://mlsanity.com/paper/2511.20462/) (2025)
- [Parallel Sampling via Counting](https://mlsanity.com/paper/2408.09442/) (2024)
- [Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition](https://mlsanity.com/paper/2410.17765/) (2024)
- [xPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding](https://mlsanity.com/paper/2608.02438/) (2026)
- [Predictability Enables Parallelization of Nonlinear State Space Models](https://mlsanity.com/paper/2508.16817/) (2025)
- [Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation](https://mlsanity.com/paper/2510.08994/) (2025)
- [S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs](https://mlsanity.com/paper/2405.20314/) (2024)

## Abstract (arXiv)

Feedforward computation, such as evaluating a neural network or sampling from an autoregressive model, is ubiquitous in machine learning. The sequential nature of feedforward computation, however, requires a strict order of execution and cannot be easily accelerated with parallel computing. To enable parallelization, we frame the task of feedforward computation as solving a system of nonlinear equations. We then propose to find the solution using a Jacobi or Gauss-Seidel fixed-point iteration method, as well as hybrid methods of both. Crucially, Jacobi updates operate independently on each equation and can be executed in parallel. Our method is guaranteed to give exactly the same values as the original feedforward computation with a reduced (or equal) number of parallelizable iterations, and hence reduced time given sufficient parallel computing power. Experimentally, we demonstrate the effectiveness of our approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of DenseNets, and (iii) autoregressive sampling of MADE and PixelCNN++, with speedup factors between 2.1 and 26 under various settings.