# Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

arXiv: https://arxiv.org/abs/2602.18312 · HTML: https://mlsanity.com/paper/2602.18312/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control](https://mlsanity.com/paper/2511.07820/) (2025)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [Simulation and Retargeting of Complex Multi-Character Interactions](https://mlsanity.com/paper/2305.20041/) (2023)
- [PDP: Physics-Based Character Animation via Diffusion Policy](https://mlsanity.com/paper/2406.00960/) (2024)
- [QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors](https://mlsanity.com/paper/2306.05666/) (2023)
- [SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrations](https://mlsanity.com/paper/2505.02094/) (2025)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [Design and Control of a Bipedal Robotic Character](https://mlsanity.com/paper/2501.05204/) (2025)
- [F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance](https://mlsanity.com/paper/2410.05791/) (2024)
- [High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures](https://mlsanity.com/paper/2504.17857/) (2025)
- [PhysicsFC: Learning User-Controlled Skills for a Physics-Based Football Player Controller](https://mlsanity.com/paper/2504.21216/) (2025)
- [Learning Linear Policies for Robust Bipedal Locomotion on Terrains with Varying Slopes](https://mlsanity.com/paper/2104.01662/) (2021)
- [Whole Body Model Predictive Control for Spin-Aware Quadrupedal Table Tennis](https://mlsanity.com/paper/2510.08754/) (2025)

## Built on by (most cited first, 2 of 2)

- [Redesigning Regularization for Effective Policy Smoothing](https://mlsanity.com/paper/2606.13169/) (2026)
- [Mind the Phase: Effective Rank and Representation Health in Legged Locomotion](https://mlsanity.com/paper/2609.06958/) (2026)

## Abstract (arXiv)

Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.