Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learning

Original: “put the green container on the bottom rack”

PDE-discovered: “grab the small green box and put it on the bottom rack”

Original promptfail

2×

PDE-discovered promptsuccess

2×

Real-world results across 4 tasks
Franka + \(\pi_{0.5}\) — success rate vs. environment steps.

PDE outperforms dense reward and action noise baselines
Across 200+ tasks in LIBERO-PRO and ManiSkill

LIBERO Pi0.5 — success rate vs. environment steps for PDE and baselines.

LIBERO Pi0 — PDE vs. action-noise baselines on the Pi0 backbone.

GR00T — PDE bootstraps where action-space exploration fails.

Bad prompts let the robot down, good prompts let the robot flySame policy weights, same scene — only the wording changes.

Refining prompts explores globally: qualitatively different strategies, not local jitter around the same wrong motor program.

Bowl: “put the green container in the bowl”

Bowl: “put the green container in the green bowl”

Drawer: “open the top drawer and put the tape inside”

Drawer: “open the top drawer and put the tape above inside”

Originalsuccess rate ≈ 5%

4×

Optimizedsuccess rate ≈ 35%

4×

Originalsuccess rate ≈ 15%

4×

Optimizedsuccess rate ≈ 45%

4×

Insight: The policy already has the motor skills for both tasks. It just doesn't know which behavior you want. The prompt is what points it at the right one: saying “green” tells the robot which bowl to aim for, and saying “above” tells it where in the drawer the tape should end up.

Try out the different prompts yourself!Click through the scenes and prompts. You can drag around and zoom in as well!

Why action noise fails: Same arm, same plan, same regretThree rollouts of an SFT VLA: action noise is trying its best, but the cabinet wins every time.

2×

The SFT checkpoint has learned a visual-scene-to-trajectory mapping and gets stuck in a single wrong motor program. Action-noise exploration jitters around it — every rollout fails in the same way, so PPO sees only zero reward and never improves.

Insight: What's needed isn't more noise but different behavior. Because the VLA is prompt-conditioned, changing the prompt changes the entire rollout, leading to global perturbations that action-space noise cannot sample-efficiently produce.

Prompt Driven Exploration (PDE)

Your VLA is stuck. Wiggling its joints harder won't help. So we hand the mic to a VLM: it watches the rollout, writes a brutally honest one-liner about what went wrong, and proposes a better prompt. Same policy weights, very different behavior.

Click any step to advance the pipeline below.

Outer loop — RL update (policy weights updated)

Inner loop — prompt refinement (frozen policy weights)

Initialize. Start with the user's task prompt \(p\) (e.g. “put the green container on the bottom rack”).
Roll out. Frozen VLA executes \(p\), producing a rollout video \(\tau\).
Evaluate. VLM watches \(\tau\) and labels it success or failure. If success, exit the inner loop.
Revise. On failure, VLM analyzes \(\tau\) and proposes a refined prompt \(p'\) (e.g. “put the green box completely on the bottom rack”) — the Update Prompt Posterior arrow.
Repeat with \(p \leftarrow p'\) until success or budget exhausted. No gradient updates on the policy.

RL update. Use the prompt pool discovered by the inner loop as a curriculum; run PPO on the original task prompt with the policy gradient flowing through the RL Update arrow back into \(\pi_\theta\).

Connecting to Posterior-Sampling RLPDE is PSRL — lifted from the VLA's parameter space to the VLM's prompt space.

Posterior-sampling RL (PSRL) maintains a distribution over policies, samples one per episode, executes it, and updates from the observed trajectory and reward. It's a clean, temporally coherent exploration recipe — but for modern VLAs, a Bayesian posterior over billions of parameters is intractable to maintain or sample from.

PDE recovers PSRL through the language interface. For a frozen VLA with parameters \(\theta\), each prompt \(p\) induces a policy \(\pi_p(\cdot \mid o) := \pi_\theta(\cdot \mid o, p)\). So a distribution over prompts \(\rho(\cdot \mid g, H)\) induces a distribution over policies — restricted to the family reachable by the VLA's language conditioning. Drawing a policy hypothesis becomes choosing a prompt.

The VLM is an implicit, amortized prompt posterior. Rather than maintaining an explicit density over natural language, PDE conditions the VLM on the rollout history \(H_i = \{(g_j, p_j, \tau_j, r_j)\}_{j \lt i}\) and queries it for the next prompt. The update is in-context — no gradient steps, no explicit Bayes rule — leveraging evidence that LLM in-context learning behaves as approximate Bayesian inference.

Bad prompts let the robot down, good prompts let the robot flySame policy weights, same scene — only the wording changes.

Try out the different prompts yourself!Click through the scenes and prompts. You can drag around and zoom in as well!

Why action noise fails: Same arm, same plan, same regretThree rollouts of an SFT VLA: action noise is trying its best, but the cabinet wins every time.

Prompt Driven Exploration (PDE)

Connecting to Posterior-Sampling RLPDE is PSRL — lifted from the VLA's parameter space to the VLM's prompt space.

PosterICRA'26 Workshop on VLA Pipelines for Real Robots & Workshop on Manipulation Robustness