mlsanityData to 14 Oct 2026

Rationally Linearized MLPs

arXiv 2610.13568 ↗First posted 9 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 84 of its 111 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

84 of 111 references.

References · 84

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2020Point Transformercited by 1,470
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  10. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  13. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  14. 2020Efficient Transformers: A Surveycited by 790
  15. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  16. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  17. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  18. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 657
  19. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  20. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  21. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  22. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  23. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  24. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  25. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  26. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 388
  27. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  28. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  29. 2015Tensorizing Neural Networkscited by 250
  30. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  31. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 239
  32. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 218
  33. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 167
  34. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  35. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  36. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
  37. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 125
  38. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121

50 most cited of 84 All 84 (JSON)

Built on by

Not cited yet.