RPX TOOLKIT

Same scene.
Different story.¶

Score a perception model on real scenes recorded before, during and after manipulation, and find the phase where it breaks.

RPX covers the paper's six tasks with its protocol, metrics and robustness scores. You write one function that calls your model or API; the toolkit downloads the data, runs the split and computes the metrics.

Score your model See the six tasks

RPX overview: today each perception task is benchmarked on its own data; RPX captures every real-world scene four ways (Clutter, Interaction, Clean and an egocentric view) and scores six tasks on them (image depth, video depth, tracking, relative camera pose, general VQA, in-context VQA) with two diagnostics, phase robustness and worst-phase quality; 100 multi-object scenes, 70 single objects scanned in 360 degrees, and a D435 RGB-D, T265 stereo and pose, and GoPro egocentric rig with masks and object questionnaires. — Perception tasks are usually benchmarked on separate datasets. RPX scores six of them on the same real scenes, before, during and after manipulation.

Try it now¶

pip install 'rpx-benchmark[hub,schemas]'
python -m rpx_benchmark.examples.benchmark_tasks --task all --smoke --output rpx-check

This runs all six evaluators on synthetic inputs in a few seconds, on CPU, without downloading anything. To score your own model on real scenes, follow Get started.

What the paper found¶

Each scene is recorded before manipulation (Clutter), while hands move objects (Interaction) and afterwards (Clean), plus an egocentric view of the interaction. Mean quality drops during Interaction, and none of the 65 evaluations is both accurate and robust.

Scene 049 in Clutter, Interaction, Clean and egocentric views with mean quality per view for video depth, tracking and VQA; Interaction is lowest. A map of all 65 evaluations by worst-phase quality and phase robustness shows none in the golden zone. — Bars: mean quality 𝒥 over all evaluated models, from the paper's Tables III–VI. Map: all 65 evaluations from the paper's Fig. 4. The four views of indoor scene 049 are not frame-synchronized.

How the toolkit fits together¶

RPX data with scene049 feeds six task model interfaces, evaluation, Phi/JEDI calculation and hardware evidence. — Your model plugs into each task through an adapter; scoring, Φ/JEDI, profiling and annotation tools sit around it. Open the SVG to zoom.

The six tasks¶

Task	Input	Evaluation
T1 Image depth	RGB image	Metric-depth accuracy
T2 Video depth	RGB sequence	Depth accuracy and temporal consistency
T3 Object tracking	Video and initialization	Multi-object tracking and identity consistency
T4 Relative camera pose	Image pairs	Relative-pose errors and threshold AUC
T5 VQA	Image and text question	Localization accuracy
T6 In-context VQA	Image, question, and reference image	Reference-conditioned localization

T5 and T6 share the visual-grounding evaluator but have separate protocols. The toolkit also has runners for detection, segmentation, sparse depth and keypoint matching; available_tasks() lists them all.

What each capture contains¶

Each exocentric capture has RGB, metric depth, instance masks, calibrated fisheye stereo and camera pose.

Scene049 Interaction RGB frame00185. — RGBAppearance and model input

Colorized metric-depth visualization of the same scene049 Interaction capture. — METRIC DEPTHGeometry and depth ground truth

Left T265 fisheye view of the scene049 Interaction capture. — FISHEYE / LEFTCalibrated stereo input

Right T265 fisheye view paired with the left view. — FISHEYE / RIGHTPaired viewpoint

Scene 049, Interaction, frame 00185. The depth image is colorized for display; scoring uses the metric values. The data guide covers each modality and the annotations.

Where to go next¶

01Run a benchmarkInstall, score a model on real scenes and read the results. 02Bring your own modelWrap a local callable or an API client. 03Six task walkthroughsInputs, outputs and commands for each paper task. 04Φ/JEDI calculatorRobustness and worst-phase quality from raw metrics. 05Hardware profilerLatency, memory and a record of the hardware. 06Mask annotation pipelineBox curation, SAM2 propagation, review and identity mapping. 07Add a metricRegister your own metric calculator. 08Extend the tasksAdd a runner, or a new task with its own inputs and scoring.

Data tools, batch adapters, visualization and the rest are listed under all capabilities.

Concepts and acronyms¶

Term	Meaning
MOS, SOS	Multi-object scenes (the 100 benchmark scenes) and single-object captures (the 70 objects, recorded on their own)
Phase	Clutter, Interaction or Clean. The egocentric recording of Interaction is a separate view.
Easy, Medium, Hard	Difficulty tiers of 33, 33 and 34 scenes, stratified by how much correction each scene's masks needed
JEDI, 𝒥, 𝒥_min	JEDI (Joint Empirical Desirability Index) combines a task's metrics into one 0–1 quality score per scene and phase. 𝒥 is its mean over all cells; 𝒥_min is the mean of the worst phase.
Φ	Phase robustness, from a repeated-measures analysis of the paired raw metrics. A model can be robust and still inaccurate, so read Φ next to 𝒥_min.
GT	Ground truth

Cite RPX¶

If RPX is useful in your work, please cite the paper; the BibTeX entry is on the citation page. The toolkit is MIT-licensed and the dataset is CC BY 4.0.

Same scene.Different story.¶