BagelVLA: Enhancing Long-Horizon Manipulation via
Interleaved Vision-Language-Action Generation
Abstract
Equipping embodied agents with the ability to reason about tasks, foresee physical outcomes, and generate precise actions is essential for general-purpose manipulation. While recent Vision-Language-Action (VLA) models have leveraged pre-trained foundation models, they typically focus on either linguistic planning or visual forecasting in isolation. These methods rarely integrate both capabilities simultaneously to guide action generation, leading to suboptimal performance in complex, long-horizon manipulation tasks. To bridge this gap, we propose BagelVLA, a unified model that integrates linguistic planning, visual forecasting, and action generation within a single framework. Initialized from a pretrained unified understanding and generative model, BagelVLA is trained to interleave textual reasoning and visual prediction directly into the action execution loop. To efficiently couple these modalities, we introduce Residual Flow Guidance (RFG), which initializes from current observation and leverages single-step denoising to extract predictive visual features, guiding action generation with minimal latency. Extensive experiments demonstrate that BagelVLA outperforms existing baselines by a significant margin on multiple simulated and real-world benchmarks, particularly in tasks requiring multi-stage reasoning.
Model Architecture
BagelVLA utilizes a Mixture-of-Transformers (MoT) architecture, comprising three independent transformers specialized for linguistic, visual, and action modalities. To tackle long-horizon tasks and semantic generalization, we formulate language-conditioned action learning as a long-sequence interleaved planning problem. These modalities are structured into a unified sequence, enabling the model to generate predictions across all three modalities based on the interleaved context.
To address the high latency in combining visual generation with control, we introduce Residual Flow Guidance (RFG). Instead of generating future frames from scratch, RFG conditions on the current observation as a strong structural prior and performs single-step denoising to predict the residual change toward the next keyframe. RFG provides a lightweight predictive visual representation that captures task-relevant dynamics with minimal overhead. This substantially reduces the computational cost of foresight while preserving its utility for action generation.
Experiment
We conduct extensive experiments on both simulated and real-world robotic tasks to evaluate the performance of BagelVLA. The simulated environments include the Calvin benchmark and Robotwin benchmark, while the real-world tasks encompass dual-arm manipulation tasks on the AgileX robot platform.
Simulation Results
| Model | Calvin ABC-D | Robotwin Clean | Robotwin Randomized |
|---|---|---|---|
| π₀ | 3.648 | 46.42 | 16.34 |
| RDT | - | 34.50 | 13.72 |
| UP-VLA | 4.078 | 52.92 | 15.16 |
| VPP | 4.329 | - | - |
| BagelVLA (Ours) | 4.405 | 75.26 | 20.87 |
Real-World Basic Tasks
| Model | Pick&Place (Seen) | Pick&Place (Unseen) | Water Flower | Stack Cubes | Stack Bowls | Sweep Rubbish | Average |
|---|---|---|---|---|---|---|---|
| π₀ | 95 | 55 | 50 | 65 | 70 | 55 | 65.0 |
| VPP | 85 | 45 | 60 | 50 | 55 | 45 | 59.5 |
| BagelVLA (Ours) | 95 | 85 | 60 | 80 | 90 | 80 | 75.5 |
Long-Horizon Planning Tasks
We designed two categories of long-horizon tasks: Stack Cubes in Requested Order and Calculate and Place Symbol Blocks. These tasks require both visual-language interleaved planning ability and instruction-following capability at the action level.
| Model | Stack Cubes (Easy/Middle/Hard) | Success Rate | Calculate & Place (Easy/Middle/Hard) | Success Rate | ||||
|---|---|---|---|---|---|---|---|---|
| π₀ | 75 | 35 | 10 | 40.0 | 70 | 25 | 0 | 31.7 |
| VPP | 60 | 15 | 0 | 25.0 | 60 | 10 | 0 | 23.3 |
| BagelVLA (Ours) | 95 | 65 | 60 | 73.3 | 80 | 65 | 45 | 63.3 |
Visualization of Interleaved Planning
Given a global instruction and the current observation, BagelVLA leverages the context to identify the immediate subtask, predicts a goal image for that subtask, and subsequently generates actions.
Demo Videos
Complex Long-Horizon Tasks
Assembling building blocks to complete the equation 24+8=?
Basic Tasks
We deployed the robot in various tasks including pick-and-place, sweeping, stacking, pressing, closing, pouring, and so on. We rollout policy with unseen objects and scenes.
Pick up the broom, sweep the garbage into the dustpan.
Place the phone into the box.
Pour the fries on the right into the pink plate on the left.
Put the mango into the brown plate.
Put the peach into the pink plate.
Put the pear into the blue plate.
Stack all the blocks.
Stack all the bowls.
Put the cup on the blue plate.
Put the hat on the brown plate.
Put the magic cube into the green plate.
Stack the orange on the pink block.
Put the red block into drawer and close it.
Press the buttons on the desktop to activate them.
Pick up the kettle, then water the flowers.