Training-Only Video Supervision

Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies

Future-video prediction improves a pretrained VLA during fine-tuning. At deployment, the video branch is removed and the original action pathway is retained.

71.18%
RoboTwin average · reference 58.33%
61.25%
Real-world success · base policy 42.50%
No video branch
Original action pathway at deployment

01 / Overview

Future video becomes a training signal.

Future frames in robot demonstrations describe object motion and task progress beyond the action labels.

We turn those observations into an auxiliary learning signal for the shared VLA representation. The video branch is used only during training.

Overview. Current RGB observations, language, and robot state are encoded into a shared vision-language prefix for the original action expert. During fine-tuning, a randomly initialized Video DiT learns to predict future video latents supplied by a frozen VAE, conditioned on adapted prefix tokens concatenated with frozen T5 features and a separate current-frame visual condition. The video loss and action loss jointly update the shared representation; the private action expert is updated only by the action loss. After training, the video expert and adapter are removed, retaining the original action-generation pathway. The evaluated random DiT-2.2B configuration achieves 71.18% average success on RoboTwin 2.0 and raises matched real-world success from 42.50% to 61.25%.

02 / Method

Train with video. Act with the VLA.

Two objectives shape a shared representation. Only the original action pathway is retained at deployment.

  1. 01 / Encode

    Build policy and visual conditions.

    A trainable adapter connects the shared VLM to video cross-attention; a VACE condition network supplies current-frame visual information.

  2. 02 / Optimize

    Combine action and video objectives.

    Flow-matching video loss is evaluated only on valid future latent blocks, alongside the original action objective.

  3. 03 / Deploy

    Remove the auxiliary expert.

    The updated shared prefix and action expert retain the original action-generation pathway.

Training and deployment. Training: the shared VLM encodes current observations, language, and state. A trainable adapter maps its prefix tokens into the video conditioning space, where they are concatenated with frozen T5 tokens. Current RGB supplies a separate VACE visual condition. The randomly initialized video expert predicts latent flow, supervised only on valid future blocks. Action and video losses both update the shared VLM; the private action expert receives only action-loss gradients. Deployment: the video expert, adapter, and frozen auxiliary encoders are omitted from action inference.
50
RoboTwin training tasks
50 / task
Clean demonstrations
50k
RoboTwin checkpoint steps
10,000
RoboTwin evaluations per configuration

03 / Results

Predictive supervision improves the deployed policy.

Improved action success after removing the video branch. Full benchmark comparisons are available below.

RoboTwin 2.0

71.18%
+12.85 ppfrom 58.33%
π0.5 reference58.33
Ours71.18

50 tasks · Clean and randomized scenes. Compared with the published base-policy reference.

Real-world manipulation

61.25%
+18.75 ppfrom 42.50%
Original π0.542.50
Ours61.25

Four tasks · 80 trials per policy. Matched demonstrations, training budget, and evaluation conditions.

RoboTwin 2.0 · LIBERO · LIBERO Plus

Random DiT-2.2B achieves 71.18% RoboTwin average success.

Method / expertRoboTwin CleanRoboTwin RandomRoboTwin AverageLIBEROLIBERO Plus
FastWAM77.761.9239.8497.6052.56
X-VLA67.9620.9544.4698.1071.40
Robust-WAM81.2034.4057.8097.9058.90
4D-WAM81.5041.8061.6598.6071.01
PrismWAM87.9853.1670.5799.3579.44
π0.5 reference70.6646.0058.3396.8584.66
π0.5 + DiT-2.2B (Random)84.8457.5271.1898.0585.15

π0.5 RoboTwin and standard LIBERO scores are official references; its LIBERO Plus score is the authors’ evaluation. RoboTwin Average equally weights Clean and Random; LIBERO averages four suites; Plus pools 10,030 scenarios. Cross-method scores follow the comparisons reported in the paper.

04 / Real world

Learn from future video. Act in the real world.

Four tasks on an AgileX ALOHA robot. Watch representative executions below.

Stack Three Bowls

Grasp and stack three bowls in sequence.

Open video ↗

Hang Cup on Mug Tree

Coordinate both arms to grasp, transfer, and hang the cup.

Open video ↗

Place Wooden Block on Towel

Transfer the wooden block and place it on the towel.

Open video ↗

Place Beverage and Lift Basket

Place the beverage in the basket, then grasp and lift the basket.

Open video ↗
Execution sequences. Six representative stages for each task, from the initial scene to task completion. These frames illustrate the manipulation sequence; aggregate success rates are reported separately.

05 / Resources

Paper and citation.

Read the paper and cite this work.

Citation

@misc{cui2026futureprediction,
  title={Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies},
  author={Hanwen Cui and Yuanyuan Li and Qing Guo and Zhida Zhang and Chao Chen and Xi Chen and Jie Cao},
  year={2026}
}