Nine tasks, five memory types

PutBackCup

Spatial · Location recall

Observe the cup’s original spot, move it aside to clean, then return it to the previously cleaned location.

SweetCup

Attribute · Hidden attribute

Prepare one sweetened and one unsweetened coffee, then select the sweetened cup after the sugar is no longer visible.

TempReport

Attribute · Attribute recall

Recall whether hot or cold water was dispensed, then press the matching button.

CanRecall

Relational · Object–container binding

After closing the cabinets, recall the requested items’ locations, retrieve them, and transport them to the stovetop.

MilkMatch

Relational · Source–target binding

Transport milk boxes in two trips, placing each on the plate matching its source tray’s color.

DrawerSearch

Single-task Progress · Search progress

Count the drawers checked while searching for the red cube, then press the answer button that many times.

SugarCount

Single-task Progress · Action counting

Add the requested number of sugar cubes to the hot chocolate, tracking successful additions before pressing completion.

HeatAndInspect

Multi-task Scheduling · Pending obligation

Inspect drawers while heating a pan, then return to turn off the stove and submit both completed task threads.

FillAndRetrieve

Multi-task Scheduling · Pending obligation

Retrieve a can while dispensing water, then return to turn off the faucet and complete both tasks.

PutBackCup: Observe the cup’s original spot, move it aside, clean the area, and return the cup to the remembered spot.

SweetCup: Add sugar to one of two coffee cups, let it dissolve out of sight, and select that same sweetened cup rather than the unsweetened one.

TempReport: Dispense hot or cold water, remember which temperature was dispensed, and press the matching temperature button.

CanRecall: Observe the requested items’ cabinet locations, close the cabinets, retrieve the items from those remembered locations, and transport them to the stovetop.

MilkMatch: Observe each milk box’s source tray color, transport the boxes in two trips, and place each box on the plate matching its source tray.

DrawerSearch: Check drawers while searching for the red cube, count the drawers checked, and press the answer button that many times.

SugarCount: Add the requested number of sugar cubes to the hot chocolate, track successful additions, and press completion after reaching the requested count.

HeatAndInspect: Heat the pan while inspecting the drawers, return to turn off the stove, and submit both completed task threads.

FillAndRetrieve: Dispense water while retrieving the can, return to turn off the faucet, and complete both task threads.

Evaluate memory at the decision, not by task length

RoboCasa-Mem benchmark overview showing a history-dependent cup-selection example, five memory categories, demonstration construction, and semantic evaluation.

Environment Memory

What happened in the world?

Task-relevant physical information that cannot be reliably recovered from the current view.

  • SpatialEarlier object locations
  • AttributeObject properties and hidden state changes
  • RelationalAssociations between entities

Process Memory

What remains to be done?

Execution state needed to choose subsequent actions and resume unfinished activities.

  • Single-task ProgressCompleted actions, search progress, and counts
  • Multi-task SchedulingPending actions across interleaved tasks

Behavior-Conditioned Evaluation

Constraints follow the robot’s own behavior

For tasks with multiple valid paths, later semantic constraints derive from earlier actions rather than a fixed target. In SweetCup, either cup may receive sugar; success requires selecting that same cup after the sugar dissolves.

SR
Task Success Rate: complete-task success.
SCR
Stage Completion Rate: average success across predefined stages.

Hierarchical Memory-Augmented Policy

HMAP separates history-dependent symbolic planning from low-level 3D-aware control. The planner supplies an executable subgoal; the controller retrieves current geometric features conditioned on that subgoal.

HMAP architecture with a Qwen3-VL high-level Entity and Process Memory Planner and a pi0.5 low-level controller using frozen VGGT-Omega geometry, cross-attention, and action modulation.
High level

Entity + Process Memory Planner

Qwen3-VL-4B-Instruct updates sparse Entity Memory and Process Memory from the instruction, previous memories, and current and earlier observations. It emits only the active language subgoal to the controller.

Low level

Subgoal-guided 3D-aware control

Built on π0.5, the controller keeps multi-view tokens from frozen VGGT-Ω separate. It uses task-conditioned queries to retrieve relevant geometry through cross-attention; learned scale and shift transformations modulate action-expert layers.

Gains concentrate in hidden-attribute recall

Simulation uses 50 held-out scene seeds per task. Tables report SR and SCR on a 0–1 scale; Avg. is the macro-average over all nine tasks.

HMAP improves average SR from 16.2% to 24.2% and Avg SCR from 0.41 to 0.47, with gains concentrated in hidden-attribute recall.

Environment Memory

ModelPutBackCupSweetCupTempReportCanRecallMilkMatch
SRSCRSRSCRSRSCRSRSCRSRSCR
ACT0.000.480.000.070.000.090.000.090.000.03
DP0.000.200.000.040.080.210.000.050.000.05
π0.50.680.850.040.270.220.570.000.240.120.47
MemER-style0.100.220.000.210.000.580.000.210.000.19
HMAP (Ours)0.700.790.100.460.860.930.000.260.160.47

Process Memory + Overall Average

ModelDrawerSearchSugarCountHeatAndInspectFillAndRetrieveAvg.
SRSCRSRSCRSRSCRSRSCRSRSCR
ACT0.000.080.040.070.000.250.000.260.000.16
DP0.000.080.140.190.000.230.000.180.020.14
π0.50.120.320.280.230.000.320.000.390.160.41
MemER-style0.000.250.080.100.000.320.000.200.020.25
HMAP (Ours)0.080.310.280.200.000.500.000.340.240.47

Research findings

Where does explicit memory help most?

The clearest gain is on TempReport, where SR rises from 0.22 to 0.86. Conditional temperature-recall success rises from 0.41 to 0.90; SweetCup conditional recall rises from 0.22 to 0.45. This supports a targeted benefit for retrieving earlier environment states, not a uniform improvement across manipulation.

What remains difficult for Process Memory?

HMAP and π0.5 perform similarly on Single-Task Progress. On HeatAndInspect, HMAP improves SCR from 0.32 to 0.50 but not final success; FillAndRetrieve also remains unsolved. Across the evaluated policies, no method completes either multi-task scheduling task.

How should the results be interpreted?

SCR localizes progress but does not fully separate memory errors from execution errors. Evaluation is primarily in simulation, the taxonomy is not exhaustive, and demonstrations contain only successful expert trajectories.

AgileX Cobot Magic Object Transport

The robot picks from a colored source container, turns toward the opposite workspace, and selects the matching target after the source leaves the camera view.

AgileX Cobot Magic Object Transport sequence showing observation, picking, moving, target observation, and placement from wrist and external views.

Citation

Download BibTeX
BibTeX
@misc{robocasa_mem,
  author = {Chen, Sihang and Ye, Chen and Chen, Wei and Lei, Xiaohan and
            Tang, Zhipeng and Zhang, Sha and Zhang, Yuan and Chen, Zixuan and
            Deng, Jiajun and Li, Houqiang and Zhang, Yanyong},
  title = {RoboCasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation}
}

Figure