Under Review
RPX at a glance. One benchmark, six tasks, the same scenes captured in every phase of a manipulation — with a single diagnostic pair (Φ phase robustness, 𝒥 worst-phase quality) for all of them.
Robots change the scenes they perceive. A hand enters the frame, objects move, and the scene at the end differs from the start. Yet the perception models on these robots are chosen from benchmarks of frozen scenes. This paper asks how much that mismatch matters.
We introduce RPX, a benchmark of 100 real scenes with 70 everyday objects, recorded with robot-grade sensors before, during, and after manipulation — ∼133K annotated frames. We evaluate 49 pretrained models zero-shot on six tasks (65 evaluations): image depth, video depth, multi-object tracking, relative camera pose, and two forms of visual question answering.
Two numbers summarize each model: phase robustness Φ from a repeated-measures MANOVA and worst-phase quality 𝒥min. The results are not subtle. Every depth and tracking model and 20 of 24 VLM evaluations change significantly across phases (p<0.001); near-range video-depth error grows 2.0–4.4× while the robot acts, every tracker falls to Φ = 0.21–0.55 on transitions through Interaction, and the best in-context grounding reaches only 𝒥min = 0.23. Relative pose alone holds steady (Φ = 0.86–0.97) — and it is the one task Interaction is not evaluated on.
Rankings also change from task to task. The model that tops a static leaderboard is not the one that keeps working once the robot moves. RPX tells them apart — and, because every failure is localized to a specific phase and scene, gives model developers the diagnostic structure to see where a model breaks, not just that it does.
Observe · Act · Verify — every scene captured in all three states.
Static, pre-interaction. The scene is arranged; the robot observes and plans.
Dynamic, mid-manipulation. Hands enter, objects move, occlusions change.
Static, post-interaction. New arrangement; the robot re-perceives to verify.
A fourth Ego stream records the head-mounted egocentric view throughout Interaction, concurrent with the exocentric camera — the same moment from two viewpoints.
Capture setup. A hand-carried D435 + T265 rig with a head-mounted GoPro for the ego stream — robot-grade sensors, not internet imagery.
Every exocentric frame carries colour, metric depth and a 6-DoF camera pose; the egocentric stream adds a second viewpoint of the same moment. Annotations are shared across both.
Colour and metric depth. The depth sensor is also the ground truth both depth tasks are scored against.
Visual-inertial odometry. The calibrated T265→D435 extrinsic gives every D435 frame a metric camera pose.
Head-mounted, recorded throughout Interaction at 6.75× the exocentric resolution. RGB only.
Instance masks and tracks for both viewpoints, plus a per-object attribute questionnaire that drives the VQA tasks.
The multi-object stream holds ∼98K frames — 100 scenes × 3 phases at ∼250 D435 frames each, plus the ego stream. The single-object stream adds ∼35K: each of the 70 objects alone in RGB-D, pose and calibrated stereo, which supply the reference images for in-context grounding.
Same scenes · same objects · same three-phase structure.
Single-frame metric and relative depth from RGB.
Temporal depth estimation from video clips with consistency metrics.
Multi-object tracking with mask and text initialization.
Camera pose estimation from image pairs (Clutter & Clean).
Text-prompted visual grounding — localize objects by attributes.
Visual grounding from a reference image — no text identity cues.
These six are what the paper reports, not what the benchmark is limited to. The toolkit registers ten tasks — object segmentation, object detection, open-vocabulary detection, sparse depth and keypoint matching among them — each declaring a primary metric that drives the phase score. Adapter interfaces cover custom-model integration, so further tasks, metrics and models can be added as the benchmark grows.
Every task, every model, reduced to the same pair — so they can be compared at all.
Evaluation pipeline. Each model's per-scene, per-phase metrics form one tensor. A repeated-measures MANOVA turns it into Φ — how little the model moves across phases — while a desirability aggregation gives 𝒥min, how good it still is in its worst phase.
Perception degrades where the robot acts — and the degradation is measurable.
Clutter → Interaction is the hardest transition for 15 of 20 depth models, and near-range video-depth error at 0.3–1.0 m — the range a robot grasps in — grows 2.0–4.4× during Interaction across all ten models. Every tracker is phase-sensitive (Φ ≤ 0.75): transitions through Interaction fall to Φ = 0.21–0.55, while Clutter ↔ Clean stays at 0.73–0.98. Relative pose is the lone exception (Φ = 0.86–0.97, no difficulty tier significant) — but it is also the one task Interaction is not evaluated on.
Seven of twelve VLMs return a syntactically valid box on ≥ 90% of in-context questions — yet the best accuracy is 21.5% Acc@0.5, against 59.3% for the same questions asked in text. Returning a valid box does not ensure correct localization; the gap suggests difficulty matching a reference image to the target instance, though this evaluation does not isolate the underlying cause.
Φ vs. 𝒥min. Each point is one of the 65 zero-shot evaluations, ringed by task. Relative pose scores high on Φ but low on 𝒥min; tracking reaches high 𝒥min at low Φ; in-context VQA clusters at the low-quality end. The thresholds are the ones chosen for the diagnostics.
Florence 2 reaches Φ = 0.94–0.98 without ever leaving the quality floor (𝒥min ≤ 0.14). VGGT makes the opposite trade — better quality (𝒥min = 0.70) but phase-sensitive (Φ = 0.77). A high Φ can simply mean uniformly bad. Read 𝒥min first, then screen with Φ.
SAM 3.1 given box prompts is the most phase-sensitive tracker (Φ = 0.50) at solid quality (𝒥min = 0.87); the same model given only a text label falls to 𝒥min = 0.72. Text-prompted GSAM2 trades the other way — the steadiest tracker (Φ = 0.75) at lower quality (𝒥min = 0.79). Swapping a VQA prompt from text to a reference image reshuffles the ranking behind Qwen3-VL 8B. A score is only interpretable alongside the prompting protocol that produced it.
Coming soon.
BibTeX coming soon
The citation will be posted here once the paper is public. In the meantime, please get in touch via Contact.
Send any comments or questions to Jishnu | Itay | Narendhiran
jishnu.p@utdallas.edu · itay.kadosh@utdallas.edu · narendhiran.v@research.iiit.ac.in
