mlsanityData to 14 Oct 2026

Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving

arXiv 2002.03629 ↗First posted 10 Feb 2020cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

Feedforward computation, such as evaluating a neural network or sampling from an autoregressive model, is ubiquitous in machine learning. The sequential nature of feedforward computation, however, requires a strict order of execution and cannot be easily accelerated with parallel computing. To enable parallelization, we frame the task of feedforward computation as solving a system of nonlinear equations. We then propose to find the solution using a Jacobi or Gauss-Seidel fixed-point iteration method, as well as hybrid methods of both. Crucially, Jacobi updates operate independently on each equation and can be executed in parallel. Our method is guaranteed to give exactly the same values as the original feedforward computation with a reduced (or equal) number of parallelizable iterations, and hence reduced time given sufficient parallel computing power. Experimentally, we demonstrate the effectiveness of our approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of DenseNets, and (iii) autoregressive sampling of MADE and PixelCNN++, with speedup factors between 2.1 and 26 under various settings.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 6; 2024: 12; 2025: 15; 2026: 14.

52 papers.

Sort:

By area:

  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  3. 2022Continuous diffusion for categorical datacited by 214
  4. 2021Autoregressive Diffusion Modelscited by 210
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  6. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 148
  7. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  8. 2023Parallel Sampling of Diffusion Modelscited by 112
  9. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 86
  10. 2024CLLMs: Consistency Large Language Modelscited by 72
  11. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 66
  12. 2024Parallelized Autoregressive Visual Generationcited by 60
38 more
  1. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  2. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  3. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 37
  4. 2024Accelerating Parallel Sampling of Diffusion Modelscited by 30
  5. 2023DreamPropeller: Supercharge Text-to-3D Generation with Parallel Samplingcited by 20
  6. 2024Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matchingcited by 18
  7. 2025Grouped Speculative Decoding for Autoregressive Image Generationcited by 17
  8. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 15
  9. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 13
  10. 2023DeepPCR: Parallelizing Sequential Operations in Neural Networkscited by 9
  11. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  12. 2025STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flowscited by 9
  13. 2024Parallel Sampling via Countingcited by 8
  14. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 8
  15. 2026xPress: Parallel Refinement for Diffusion Drafters in Speculative Decodingcited by 8
  16. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  17. 2025Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generationcited by 7
  18. 2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
  19. 2025A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systemscited by 4
  20. 2025Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generationcited by 4
  21. 2026The Coupling Within: Flow Matching via Distilled Normalizing Flowscited by 3
  22. 2025PCM : Picard Consistency Model for Fast Parallel Sampling of Diffusion Modelscited by 2
  23. 2025Accelerate TarFlow Sampling with GS-Jacobi Iterationcited by 2
  24. 2025SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generationcited by 2
  25. 2026Parallel Jacobi Decoding for Fast Autoregressive Image Generationcited by 2
  26. 2026Accelerating Speculative Diffusions via Block Verificationcited by 2
  27. 2021PixelPyramids: Exact Inference Models from Lossless Image Pyramidscited by 1
  28. 2021ELF: Exact-Lipschitz Based Universal Density Approximator Flowcited by 1
  29. 2025Accelerating Inference of Discrete Autoregressive Normalizing Flows by Selective Jacobi Decodingcited by 1
  30. 2026SJD-PV: Speculative Jacobi Decoding with Phrase Verification for Autoregressive Image Generationcited by 1
  31. 2026SJD-VP: Speculative Jacobi Decoding with Verification Prediction for Autoregressive Image Generationcited by 1
  32. 2026SNLP: Layer-Parallel Inference via Structured Newton Correctionscited by 1
  33. 2025Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attentioncited by 0
  34. 2026The Context-Ready Transformercited by 0
  35. 2026CSD: Content-aware Speculative Decoding for Efficient Image Generationcited by 0
  36. 2026Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformerscited by 0
  37. 2026SJD-SV: Speculative Jacobi Decoding with Semantics Verification for Autoregressive Image Generationcited by 0
  38. 2026GradLev: Token-Parallel Test-Time Training Via Costate Predictioncited by 0

50 most cited of 52 All 52 (JSON)