Correct Me If I'm Wrong

Language-guided exploration for VLA post-training.
Sunshine Jiang1, John Marangola1, David Zhang1, Nitish Dashora1, Pulkit Agrawal1, Zhang-Wei Hong1,2
1Massachusetts Institute of Technology  ·  2MIT-IBM Watson AI Lab
MIT MIT-IBM Watson AI Lab
📄 Paper 💻 Code — coming soon

By doing real-world RL with language intervention, a VLA can learn a zero-success-rate task in under an hour.

2×
A training rollout with mid-rollout language intervention: the operator watches the robot, and when it fumbles, updates the prompt (“pull open the top drawer” → “move the bowl higher” → “put the bowl inside”). The VLA keeps generating actions — only its language input changes.

Why is RL sample inefficient?RL keeps failing in the same way.

2×
RL only learns from what it collects, and the same repeated failures teach it virtually nothing.

Failures happen, and humans can't do anything about it

2×
Instead of giving the policy hints, when the policy gets stuck the human watching can only reset and wait for the policy to try again.

Intervention helps RL escape from the same failure — but doesn't scaleTask: open the top drawer and put the bowl inside.

2×
Intervening on robot control requires experienced operators. A SpaceMouse correction takes ~30s and an expert to place the end-effector precisely — novices drag the policy through failure rather than teaching it a better strategy.

Language intervention works and scalesTask: open the top drawer and put the bowl inside.

Correct a high-level semantic failure

2×
The policy targets the wrong object or sub-goal; a short prompt naming the target (e.g. “pull open the top drawer”) re-routes it.

Correct a low-level motor control failure

2×
The target is correct but execution is off — a motor-cue prompt (e.g. “move the bowl higher”) tightens approach angle and contact height without any weight update.

Different operators, same result

2×
Three operators of varying robotics experience issue their own corrections on the same checkpoint. The success rate is robust to operator choice — the policy is not overfit to any one person's phrasing.

Guided by intervention, learn to succeed without interventionInstruction-Driven Exploration (IDE): human re-writes the prompt mid-rollout; PPO backprops through both intervened and canonical prompts.

Your VLA is stuck on the canonical prompt. Wiggling its joints won't help. So we hand the mic to the human already standing there: they watch the rollout, and when the robot goes sideways, they update the prompt mid-rollout. The VLA keeps generating actions; only its language input changes. Then we fold those successes back into PPO — but reward is still scored against the original task.

The pipeline animation: a rollout on the canonical prompt \(p_g\) is redirected mid-way by interventions \(p_4, p_{20}\) into a successful trajectory. IDE then does a mixed PPO update — the policy gradient flows through both the intervened prompts (teal path) and the original prompt (purple path), so exploration comes from \(p_t\) but the deployed policy stays aligned with \(p_g\).

Two knobs make it practical: mixed backprop (so the policy doesn't become dependent on the human's phrasing), and adaptive intervention (allow interventions in proportion to \(1 - \hat s(g)\) — low-success tasks get more help; reliable ones get left alone).

IDE improves sample efficiencyConsistently beats PPO.

By doing real-world RL with language intervention, a VLA can learn a zero-success-rate task in under an hour.

Task: open the top drawer and put the bowl inside. PPO stays near zero — it never collects a successful rollout. IDE reaches non-trivial success within the first iteration and keeps improving.

IDE improves sample efficiency4× the performance of action intervention (DAgger).

Per-task success rate (%) after RL fine-tuning; (+X) is the gain over the SFT checkpoint under the canonical prompt. Action intervention is implemented as DAgger.
Task SFT PPO DAgger PPO + IDE (ours)
put the tape in the bowl 20.030.0 (+10)30.0 (+10) 75.0 (+55)
put the tape in front of the camera 0.015.0 (+15)35.0 (+35) 60.0 (+60)
put the donut in the bowl 5.020.0 (+15)10.0 (+10) 55.0 (+50)
open the top drawer and put the bowl inside 0.00.0 (+0)10.0 (+10) 35.0 (+35)
Average 6.316.3 (+10)22.5 (+16.2) 56.3 (+50.0)
A typed prompt takes ~2s; a SpaceMouse correction takes ~30s. The operator is in active control for ~5% of IDE training vs. ~75% under DAgger — for a policy that reaches 4× the average success rate.

IDE improves robustnessRobust recovery from failure.

2×
The IDE-trained policy recovers from a wider range of failure modes than the baselines — it saw diverse prompts during training, so at test time it stays responsive to language corrections instead of collapsing to a single memorized trajectory.

Practical real-world RL under a tight data budget.

Abstract

Reinforcement learning is a promising framework for improving vision-language-action (VLA) models through interaction, but it often struggles when the initial policy repeatedly fails in the same way. Our key observation is that these models already offer a simple way for humans to change what the policy attempts — through language. In real-world robot training, humans often need to stay nearby to monitor rollouts, reset environments, and check progress; while watching, a human can redirect exploration by simply revising the instruction, without providing low-level actions or directly controlling the robot. We introduce Instruction-Driven Exploration (IDE), an interactive post-training framework that uses human prompt updates to help VLAs collect more useful experience. On real-world physical robots, IDE significantly improves sample efficiency over standard RL, making robot learning more practical under limited data. We further show that interactive post-training makes VLAs more responsive to prompt updates at test time, enabling users to guide robot behavior through natural language during execution.

Applicable to diverse objects and layoutsIDE improves generalization — PPO and DAgger do not.

2×
OOD banana on the put the tape in the bowl task. Given the same mid-rollout correction, only IDE re-routes to the banana and completes the deposit; PPO drives toward an OOD end-effector pose, and DAgger's recovery is bound to its narrow training scene.