mlsanityData to 7 Oct 2026

Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation

arXiv 2209.10780 ↗First posted 22 Sep 2022cs.RO cs.AI cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied public spaces. To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the robust handling of system constraints from Model Predictive Control (MPC). Our approach, called Performer-MPC, uses a learned cost function parameterized by vision context embeddings provided by Performers -- a low-rank implicit-attention Transformer. We jointly train the cost function and construct the controller relying on it, effectively solving end-to-end the corresponding bi-level optimization problem. We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in different challenging real-world scenarios. Compared with a standard MPC policy, Performer-MPC achieves >40% better goal reached in cluttered environments and >65% better on social metrics when navigating around humans.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 60 of its 100 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

60 of 100 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
38 more
  1. 2016End to End Learning for Self-Driving Carscited by 1,000
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2019Differentiable Convex Optimization Layerscited by 458
  9. 2021Implicit Behavioral Cloningcited by 442
  10. 2021A Survey of Transformerscited by 399
  11. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  12. 2021Efficient and Modular Implicit Differentiationcited by 199
  13. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  14. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  15. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  16. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  17. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
  18. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  19. 2021Core Challenges of Social Robot Navigation: A Surveycited by 97
  20. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  21. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 64
  22. 2021Improved Transformer for High-Resolution GANscited by 59
  23. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  24. 2020Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Surveycited by 48
  25. 2016From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robotscited by 37
  26. 2019Learning Convex Optimization Control Policiescited by 31
  27. 2019Differentiable Gaussian Process Motion Planningcited by 27
  28. 2021Learning Inverse Kinodynamics for Accurate High-Speed Off-Road Navigation on Unstructured Terraincited by 23
  29. 2022VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamicscited by 19
  30. 2020APPLD: Adaptive Planner Parameter Learning from Demonstrationcited by 17
  31. 2021Visual Representation Learning for Preference-Aware Path Planningcited by 17
  32. 2020APPLI: Adaptive Planner Parameter Learning From Interventionscited by 16
  33. 2021APPL: Adaptive Planner Parameter Learningcited by 15
  34. 2021APPLE: Adaptive Planner Parameter Learning from Evaluative Feedbackcited by 13
  35. 2022A Protocol for Validating Social Navigation Policiescited by 12
  36. 2020APPLR: Adaptive Planner Parameter Learning from Reinforcementcited by 11
  37. 2020Toward Agile Maneuvers in Highly Constrained Spaces: Learning from Hallucinationcited by 10
  38. 2020Fitting a Linear Control Policy to Demonstrations with a Kalman Constraintcited by 9

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 11; 2025: 4; 2026: 2.

27 papers.

Sort:

By area:

  1. 2023SACSoN: Scalable Autonomous Control for Social Navigationcited by 60
  2. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  3. 2023Principles and Guidelines for Evaluating Social Robot Navigation Algorithmscited by 52
  4. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
  5. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  6. 2023Simplex Random Featurescited by 9
  7. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  8. 2023Graph Reinforcement Learning for Network Control via Bi-Level Optimizationcited by 7
  9. 2024DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learningcited by 5
  10. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  11. 2024Grounded Curriculum Learningcited by 5
  12. 2022A Contextual Bandit Approach for Learning to Plan in Environments with Probabilistic Goal Configurationscited by 4
15 more
  1. 2023A Study on Learning Social Robot Navigation with Multimodal Perceptioncited by 4
  2. 2024Learning Multiple Initial Solutions to Optimization Problemscited by 4
  3. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  4. 2025GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoringcited by 3
  5. 2022Robot Navigation with Reinforcement Learned Path Generation and Fine-Tuned Motion Controlcited by 2
  6. 2024Driving from Vision through Differentiable Optimal Controlcited by 2
  7. 2024Learning Social Cost Functions for Human-Aware Path Planningcited by 2
  8. 2023GAN-MPC: Training Model Predictive Controllers with Parameterized Cost Functions using Demonstrations from Non-identical Expertscited by 1
  9. 2024Mapping back and forth between model predictive control and neural networkscited by 1
  10. 2025MATT-Diff: Multimodal Active Target Tracking by Diffusion Policycited by 1
  11. 2024Dyna-LfLH: Learning Agile Navigation in Dynamic Environments from Learned Hallucinationcited by 0
  12. 2024Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problemcited by 0
  13. 2025TamedPUMA: safe and stable imitation learning with geometric fabricscited by 0
  14. 2026APPLV: Adaptive Planner Parameter Learning from Vision-Language-Action Modelcited by 0
  15. 2026RAY-TOLD: Ray-Based Latent Dynamics for Dense Dynamic Obstacle Avoidance with TDMPCcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning
  2. 2024Faster Model Predictive Control via Self-Supervised Initialization Learning
  3. 2025Bootstrapped Model Predictive Control
  4. 2025Model Predictive Adversarial Imitation Learning for Planning from Observation
  5. 2024Navigating the Human Maze: Real-Time Robot Pathfinding with Generative Imitation Learning
  6. 2026NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation
  7. 2018Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations
  8. 2026Less Is More: Scalable Visual Navigation from Limited Data