Visual Buffer
Recent frames arrive as a stream. The buffer holds enough context for the next update, then clears when evidence is committed.
observe compress
Streaming reasoning for executable memory in robot control
Plan first.
Act later.
PAL turns a stream of visual evidence into semantic events and a future-ready subtask list, so a robot can use what it saw before it needs to act.
56.05%RoboMME average success
+11.54 ppover strongest baseline
60.83%real-world overall success
01 / The problem
The robot has seen it. The robot cannot see it now.
Long-horizon manipulation often hides the evidence that determines the next action. A cube is covered early in a video, a pattern is demonstrated once, or progress must be counted across repeated picks.
A short visual window can execute a motion. It cannot reliably recover an event that has already left the frame.
PAL makes future-relevant evidence executable before the moment of action arrives.
VideoUnmask · simulation · same episode, frames 0 and 32
02 / The idea
Three memory layers. One executable state.
PAL separates recent pixels, remembered events, and actions that are ready to run. Each layer has a different time scale and a different job.
Event Memory
Task-relevant changes become compact natural-language events: what moved, where it ended, and why it matters later.
remember ground
Subtask List
Future subtasks are appended before execution. The low-level VLA selects the next item without searching the whole visual history.
plan act
03 / Interactive walkthrough
Watch the memory become a plan.
In VideoUnmask, the green cube disappears under a container. Its remembered position determines what the robot must lift.
Conditioning video · frame 0OBSERVE
The green cube is visible before the containers cover it.Visual buffer
- Frame 0 · colored cubes visible
Event memory
- No committed event yet
Subtask list
- No executable subtask yet
The conditioning video reveals the cube's identity and position before that evidence becomes hidden.
Current actionWatch the conditioning video.
VideoUnmask · simulation · episode 2 · camera crops; memory and action labels abbreviated from the rollout.
04 / Evidence
Longer memory, stronger control.
Across memory-dependent RoboMME tasks, PAL improves most when the relevant evidence arrives long before execution.
56.05%PAL average on RoboMME
+11.54 ppversus FrameSamp-Modul, the strongest baseline
81.60%long-video success, where streaming updates matter most
| Method | Short-video | Long-video | In-context | Episode-start |
|---|---|---|---|---|
| FrameSamp-Modul | 41.87% | 46.00% | 27.05% | 72.96% |
| MemER | 37.52% | 28.00% | 41.00% | 65.11% |
| PAL | 45.26% | 81.60% | 46.20% | 77.33% |
Training the timing of memory.
Dense supervision teaches updates; task-grounded self-labeling refines when to commit them. Both stages contribute to the final result.
Stage 1 only37.35%
Dense supervision alone can update the executable list too frequently.
Sparse label43.39%
Sparse supervision can miss updates needed across a long video.
PAL · two-stage training56.05%
Verified self-labels improve the timing and reliability of memory updates.
04a / Simulation evidence
Remember the count, the order, the hidden object.
Three successful RoboMME rollouts from the main evaluation. Paired camera views show the evidence the robot observes and the actions it takes.
PickXtimes
Pick and place the green cube three times, then press the stop button.
simulation · episode 26 · successful rolloutRollout overview
Event Memory
- No separate event updates. The subtask list encodes the repeat count.
Subtask List
- Pick up cube · first time
- Place cube on target
- Pick up cube · second time
- Place cube on target
- Pick up cube · third time
- Place cube on target
- Press stop button
No executable subtask yet.
VideoPlaceOrder
Return the green cube to the second target visited in the demonstration.
simulation · episode 21 · successful rolloutRollout overview
Event Memory
- The cube's second placement identifies the destination, even after it moves again.
Subtask List
- Pick up the green cube
- Place cube on its second target
No executable subtask yet.
VideoUnmask
Lift the container hiding the green cube using its remembered position.
simulation · episode 2 · successful rolloutRollout overview
Event Memory
- The green cube's position is retained when the containers conceal it.
Subtask List
- Lift the container hiding the green cube
No executable subtask yet.
Event memory and subtasks abbreviated from each rollout's prediction and execution logs.
05 / Real world
The same memory, on a Franka arm.
Four tasks test different kinds of long-horizon recall: count progress, recover a hidden layout, remember an earlier placement, and reproduce a dense pattern.
Complete recorded demos at 4× source playback. The success rates summarize the evaluation trials.
Real-world demo4× source playback
