NeurIPS 2026

Prompt-Driven Exploration

Language as an Exploration Space
for VLA Reinforcement Learning

Sunshine Jiang1,3 John Marangola1,3 David Zhang1,3 Raghuram Kowdeed3 Ruiyang Luo1,3
1 MIT2 MIT-IBM Computing Research Lab3 Improbable AI Lab

More samples or better prompts?

VLA RL can spend millions of steps without finding success.

Both methods start at 0%

PPO

× Lifts the mug “Close the microwave”

PPO + PDE

× Lifts the mug “Close the microwave”

Success rate over the first 10 million environment steps Both methods start at zero with the original prompt. A lighter segment connects the shared start to the first plotted PDE value as a visual guide. The solid traces show approximate rates from Figure 3. PPO + PDE reaches about 52 percent at 5 million steps and 77 percent at 10 million. PPO stays near zero. The videos are selected fixed-policy rollout examples. 100% 50% 0% 0 5M 10M RL environment steps Illustrative connection to the shared start Both methods start at 0%

PDE doubles real-robot success

>2×PPO’s success rate
128training rollouts
≈1 hof robot data

Real-robot success rate

Real-world learning curvesAcross checkpoints at 0, 64, 128, and 192 training rollouts, PPO plus PDE reaches success rates of 10, 30, 30, and 50 percent. PPO reaches 10, 10, 15, and 15 percent.

Why does RL need so many samples?

Action noise

“Open the drawer and put the bowl inside”

Prompt exploration

“put the green container on the bottom rack”

Prompt edits steer unpredictably

Loading the recorded prompt responses…

Prompt-Driven Exploration (PDE)

Watch. Reflect. Refine.

The VLM revises prompts from rollout feedback, and PPO updates the VLA.

The VLM sees that the robot lifts the mug and revises the prompt to specify the door. The VLA tries the new prompts, and PPO updates the policy from rollout experience.

Posterior sampling through prompts

Posterior sampling

Sample a world model and follow its optimal policy.

Mk∼P(M∣Hk) πk∈argmaxπJMk(π)

PDE

Sample a prompt and follow the VLA policy it induces.

pk∼ρVLM(·∣g,Hk) πk(·∣o)=πθ(·∣o,pk)

Prompt search objective

p⋆∈argmaxp 𝔼[R(τ,g)∣p,πθ]

Both commit to one policy for the episode and use feedback to guide the next sample.

Simulation benchmarks

On LIBERO-PRO, PDE is 1.7–2.5× as sample-efficient as PPO.

LIBERO-PRO

120 task instances

47 hard tasks with 1.3% mean initial success

Robot placing a burger onto a plateRobot placing a bottle onto a plate

ManiSkill

12 test variants · 25 objects

Hard tasks start below 25% success

Task difficulty

LIBERO-PRO · 47 hard tasks · π₀.₅

Below 10% success before RL

Mean ± SE

More real-world examples

Bottom rack

Original prompt“put the green container on the bottom rack”
PDE prompt“grab the small green box and put it on the bottom rack”

Green bowl

Original prompt“put the green container in the bowl”
PDE prompt“put the green container in the green bowl”

Top drawer

Original prompt“open the top drawer and put the tape inside”
PDE prompt“open the top drawer and put the tape above inside”
Interactive 3D

Take a closer look.

Replay a recorded behavior. Move the camera. Compare prompts.

π0.5 · LIBERO-PRO
NeurIPS 2026

Prompt-Driven Exploration

Language as an Exploration Space for VLA Reinforcement Learning

BibTeX
@inproceedings{jiang2026promptdriven,
  title={Prompt-Driven Exploration: Language as an Exploration Space for {VLA} Reinforcement Learning},
  author={Jiang, Sunshine and Marangola, John and Zhang, David and Kowdeed, Raghuram and Luo, Ruiyang and Dashora, Nitish and Li, Richard and Agrawal, Pulkit and Hong, Zhang-Wei},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2026},
  url={https://arxiv.org/abs/2607.08837}
}