Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

Abstract

Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions.

A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy.

On continuous in-hand rotation with a 16-DoF Allegro hand, the method reaches 100% success in policy-only evaluation (5/5 trials) after 7 minutes of online RL, following initialization with 20 MPC trajectories collected on hardware in 12 minutes. Online training incurs about three object drops on average. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller. It completes 1000 consecutive rotations over more than 110 minutes without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance. We further demonstrate rapid adaptation to different object geometries and successful goal-conditioned reorientation, showing that the framework enables efficient, low-intervention, real-world dexterous RL.

Method

Overview of our framework. Stage 1 (Offline MPC Data Collection): trajectories are collected via MPC. Stage 2 (Pretraining): the collected demonstrations pretrain the actor using behavior cloning with standard-deviation initialization and the critic via Soft-SARSA. Stage 3 (Online SAC with Mixed Replay and MPC-Policy Scheduling): the actor-critic network is fine-tuned with SAC using a mixed replay buffer combining offline demonstrations and online transitions. An MPC-policy scheduling mechanism dynamically alternates control between the learned policy and MPC for guided exploration.

Sampling-based MPC serves as a scaffold for an off-policy learner, and the learned policy progressively replaces MPC during real-world interaction. Both MPC and the policy command the same joint-level hand-control interface, so the off-policy learner uses their transitions identically, regardless of which controller generated them.

Real-World Results

Real-World Learning

The three roles of MPC are complementary

MPC replay (R) preserves task-relevant prior experience, pretraining (P) provides a better starting point, and online guidance (G) improves the quality and consistency of newly collected interaction.

Ablation results for continuous in-hand rotation.

Adaptation to new object geometries

Starting from the policy trained on the 45 mm hexagonal object, continued real-world learning with online MPC guidance adapts it to a 35 mm hexagonal object and a 55 mm circular object, without new offline MPC data or repeated pretraining.

35 mm hexagonal object

55 mm circular object

Goal-conditioned reorientation

The same framework learns goal-conditioned reorientation about an axis perpendicular to the object’s principal axis, reaching commanded headings within ±20° while maintaining the grasp.

Implementation Details

Real-world SAC with MPC scaffolding

Parameter	Value
Control	10 Hz joint position targets for 16 Allegro DoF, episode horizon 900 steps
Observation / action dimension	60 / 16
Actor network	MLP 60 → 256 → 256 → (16 mean, 16 log-std), tanh activations, LayerNorm; tanh-squashed Gaussian, std clipped to [10⁻⁵, 5]
Critic network	Ensemble of 10 Q-nets, each (60 + 16) → 256 → 256 → 1, tanh, LayerNorm; 2 randomly subsampled per target (REDQ-style)
Discount γ	0.995
Learning rates (Adam)	actor 3×10⁻⁴ (linear warm-up over 200k updates), critic 3×10⁻⁵, temperature 10⁻⁴
Batch size	256
UTD ratio	8
Polyak τ	0.01
Temperature α	auto-tuned, initial 0.05, target entropy −16 (= −action_dim × 1.0); entropy backup on
α_eff (critic pretraining)	0.02
BC std	mean-only BC (std-loss weight 0), then log-std head set to constant std = 0.3 after BC
Action-clip ε	0.02 (demo actions clipped to [−1 + ε, 1 − ε] before the pre-tanh BC target)
BC gradient steps	12,000 (lr 3×10⁻³, batch 256)
Soft-SARSA gradient steps	50,000 (lr 3×10⁻⁵, batch 256, recorded next action, CQL off)
Demonstrations (rotation)	20 hardware MPC episodes, one full rotation each, ~12 min
Demonstrations (reorientation)	~7 min of hardware MPC interaction
Replay buffer / offline ratio	capacity 10⁶, offline sample ratio 0.3
MPC-policy mixing	policy probability 0.5 → 1.0 over 6,000 env steps (~10 min), chunk length 40
Rotation reward	λ_ψ = 20, λ_τ = −0.3, b_succ = 200, b_drop = −50
Evaluation	5 policy-only episodes, 90 s horizon, success = one full 2π rotation without a drop

Soft-SARSA pretraining used the same γ, τ, critic ensemble, and Adam optimizer as online SAC, with learning-rate warm-up disabled.

PPO sim-to-real baseline

Stable-Baselines3 2.8.0; unlisted PPO settings are SB3 defaults.

Parameter	Value
Parallel envs	192 (subprocess)
Rollout per update	128 steps × 192 envs = 24,576
Minibatch / epochs	8,192 / 10
Learning rate	3×10⁻⁴ constant (Adam)
γ / GAE λ	0.99 / 0.95
Clip range	0.2, advantage normalization on
Entropy coef / value coef	0.01 / 0.5
Max grad norm	0.5
Total timesteps	15M
Env	10 Hz control, 900-step episodes, action EMA α = 0.5
Domain randomization (uniform per episode reset; noise Gaussian per step)
Object size scale	U[1.0, 1.05] (mass and inertia rescaled with size)
Object mass	U[0.15, 0.20] kg
Sliding friction	object U[0.3, 0.5]; fingertips U[0.5, 0.8]; other hand geoms U[0.1, 0.3]
Allegro PD gain scale	U[0.6, 1.2]
Initial object position	±0.01 m in y
Observation noise σ (per step)	object pos 0.01 m, quat 0.04, joint pos 0.05, joint vel 0.2, object vel 0.04, yaw 0.05, last action 0.05
Action noise σ	0.02
External force perturbation	off