API reference

Start from what you want to do, or browse by module further down. Each module page lists its classes and functions with signatures, types and source, generated from the same code as the PyPI package.

Start with a workflow

Goal	Entry points	Guide
Load and download RPX	hub.load, fetch_manifest, download_split, RPXDataset	Get started
Connect a model	BenchmarkableModel, adapters and NumPy factories	Bring your own model
Run a task	Task runners and configurations, benchmark runner	Task walkthroughs
Compute or extend metrics	Metric implementations, metric registry	Add metrics
Use VQA or grounding contracts	VQA tools, robot VQA	Model integration
Analyze robustness and efficiency	Deployment, Phi, JEDI, profiler	Concepts
Measure hardware performance	Profiler	Hardware guide
Calculate Φ/JEDI from raw metrics	Raw calculator	Analysis guide

Public Python interface

Most users only need import rpx_benchmark as rpx; the table above shows where each piece lives. api defines samples, task identities, ground truths, and predictions. The modules below cover the rest.

import numpy as np
import rpx_benchmark as rpx

# Small integration fixture, not a pretrained model.
model = rpx.make_numpy_depth_model(
    lambda rgb: np.full(rgb.shape[:2], 2.0, dtype=np.float32)
)
# Point manifest_path at your task manifest before running inference.
config = rpx.MonocularDepthRunConfig(
    model=model,
    manifest_path="/data/manifests/monocular_depth/easy.json",
    device="cpu",
    output_dir="rpx_results/my-depth/easy",
)

Installation and optional dependencies

python -m pip install 'rpx-benchmark[hub,schemas]'

The evaluators need only NumPy and SciPy. The extras add the rest: [hub] for downloads and Parquet manifests, [schemas] for manifest validation, [hf-datasets] for the Hugging Face Datasets bridge, [analysis] for the statistics, and [ar-pose-audit] for marker audits. Model dependencies are never installed by the toolkit. A module that needs an extra raises an error when called without it.

Core harness and analysis

Module	Purpose
rpx_benchmark.api	Core task enums, data contracts, and the base model interface.
rpx_benchmark.ar_pose_audit	AR-board consistency audit for RPX single-object camera poses.
rpx_benchmark.box_upload	Box upload helpers used by the task runners and post-hoc tools.
rpx_benchmark.cell_log	Per-scene, per-phase metric log: the record every analysis starts from.
rpx_benchmark.cleanup	Clean shutdown handling for long-running scripts.
rpx_benchmark.cli_ux	Colored logging and progress bars for the command-line scripts.
rpx_benchmark.decode_contracts	Canonical decode helpers with runtime contracts.
rpx_benchmark.deployment	Deployment-readiness metrics for RPX: TS, STR, SGC, ESD, weighted scoring.
rpx_benchmark.determinism	Deterministic seeding for reproducible benchmark runs.
rpx_benchmark.evaluators	Legacy metric module (compat shim).
rpx_benchmark.exceptions	Exception hierarchy for the RPX benchmark toolkit.
rpx_benchmark.hub	HuggingFace Hub integration for RPX benchmark.
rpx_benchmark.jedi	JEDI: Joint Empirical Desirability Index.
rpx_benchmark.loader	Data loading utilities for RPX benchmark.
rpx_benchmark.logging_utils	Structured logging for the RPX benchmark toolkit.
rpx_benchmark.model_profiler	Unified model profiler for all RPX tasks.
rpx_benchmark.paper_depth_analysis	Paper-level statistical analysis for the six-metric RPX D1-F cell log.
rpx_benchmark.paper_tracking_analysis	Paper-level statistical analysis for RPX D3 tracking cells.
rpx_benchmark.phi	State-Transition Robustness Φ via repeated-measures MANOVA.
rpx_benchmark.phi_jedi_summary	Glue between the runner's per-sample metrics and Φ / JEDI.
rpx_benchmark.pose_conventions	Coordinate conventions shared by RPX camera-pose benchmarks.
rpx_benchmark.pose_metrics	Relative camera pose metrics, standard and RPX-specific.
rpx_benchmark.pose_pairs	On-the-fly deterministic pose-pair generation for RPX-RCPE.
rpx_benchmark.profiler	Hardware-agnostic model efficiency profiling for RPX.
rpx_benchmark.reports	Result serialisation: JSON + markdown summary for benchmark runs.
rpx_benchmark.runner	Benchmark runner orchestrating model, data, and metrics.
rpx_benchmark.schemas	Strict, typed manifest schemas for the RPX benchmark.
rpx_benchmark.temporal_budget_sweep	Frame-budget sweep runner and degradation analysis for video tasks.
rpx_benchmark.video_loader	Sequence-shaped loader for video tasks (Video Depth today; future video tasks).

Model adapters

Module	Purpose
rpx_benchmark.adapters	Formal adapter framework for bringing models into the RPX benchmark.
rpx_benchmark.adapters.base	Core types for the RPX adapter framework.
rpx_benchmark.adapters.batched_depth	True-batched depth model wrapper.
rpx_benchmark.adapters.batched_multimodal	Generic batched-dispatch BenchmarkModel base for multi-modal tasks.
rpx_benchmark.adapters.depth	Depth-model adapters (paper task Image Depth: MONOCULAR_DEPTH).
rpx_benchmark.adapters.depth.skeletons	Skeleton adapter classes for every Image Depth (monocular depth) model.
rpx_benchmark.adapters.depth_scaffold	Shared scaffold for depth-model adapters (Image Depth and Video Depth).
rpx_benchmark.adapters.video_depth	Video-depth adapters (paper task Video Depth: VIDEO_DEPTH).
rpx_benchmark.adapters.video_depth.skeletons	Skeleton adapter classes for every Video Depth model.

Data loading and feature extraction

Module	Purpose
rpx_benchmark.data	HuggingFace datasets integration for RPX.
rpx_benchmark.data.clip_dataset	Per-scene, per-phase clip iteration for video tasks.
rpx_benchmark.data.esd	Effort-Stratified Difficulty (ESD) feature extraction.
rpx_benchmark.data.esd_scoring	ESD scoring: turns the 27-feature table into a difficulty score per scene and phase, and Easy/Medium/Hard tiers.
rpx_benchmark.data.features	Canonical datasets.Features definitions for every RPX task.
rpx_benchmark.data.hf_bridge	Row-level bridge between datasets.Dataset and :class:Sample.
rpx_benchmark.data.load	One-line datasets.load_dataset entry point for RPX.
rpx_benchmark.data.torch_dataloader	PyTorch DataLoader adapter for RPX datasets.

Dataset publishing and storage

Module	Purpose
rpx_benchmark.dataset_hub	Selective Hugging Face download and upload pipeline for the dataset.
rpx_benchmark.dataset_hub.cli	python -m rpx_benchmark.dataset_hub.cli: dataset hub sub-commands.
rpx_benchmark.dataset_hub.croissant	Stage the Croissant metadata JSON into the upload tree.
rpx_benchmark.dataset_hub.dataset_card	Generate the HuggingFace dataset card (README.md) at the repo root.
rpx_benchmark.dataset_hub.downloader	Selective download from the RPX HuggingFace dataset repo.
rpx_benchmark.dataset_hub.ego_layout	Arrange raw ego (GoPro) captures into the layout scan_capture_root expects.
rpx_benchmark.dataset_hub.lossless_convert	Re-encode a capture tree into denser lossless formats for HF upload.
rpx_benchmark.dataset_hub.manifest	Build the per-frame Parquet manifest the dataset hub ships.
rpx_benchmark.dataset_hub.mock	Synthetic mock generator that mimics the on-disk capture layout.
rpx_benchmark.dataset_hub.packer	Pack an on-disk capture tree into per-modality tar shards for HF upload.
rpx_benchmark.dataset_hub.recipes	Task → modality recipes for selective dataset downloads.
rpx_benchmark.dataset_hub.scanner	Walk an on-disk RPX capture tree and report structured inventory.
rpx_benchmark.dataset_hub.split_manifests	Per-task, per-split manifest writer for the dataset hub.
rpx_benchmark.dataset_hub.staging	Staging helpers for the non-tar files at the dataset repo root.
rpx_benchmark.dataset_hub.uploader	Upload a packed staging directory to a HuggingFace dataset repo.

Metric implementations

Module	Purpose
rpx_benchmark.metrics	Pluggable metric registry for RPX benchmark tasks.
rpx_benchmark.metrics.depth	Monocular absolute depth metric calculators.
rpx_benchmark.metrics.depth_alignment	Per-frame alignment of a predicted depth map to GT scale.
rpx_benchmark.metrics.depth_paper	Canonical RPX D1-F paper metrics and fixed camera calibration.
rpx_benchmark.metrics.depth_robotics	Robotics-deployment depth metric calculators.
rpx_benchmark.metrics.depth_temporal	Clip-level temporal depth metrics for Video Depth.
rpx_benchmark.metrics.detection	Object detection + open-vocab detection metric calculators.
rpx_benchmark.metrics.grounding	Visual grounding metric calculators.
rpx_benchmark.metrics.keypoints	Keypoint correspondence metric calculators.
rpx_benchmark.metrics.pose	Relative camera pose metric calculators.
rpx_benchmark.metrics.registry	Core of the metric plugin system.
rpx_benchmark.metrics.segmentation	Instance/semantic segmentation metric calculators.
rpx_benchmark.metrics.sparse_depth	Sparse depth metric calculators (RGB + sparse GT depth points).
rpx_benchmark.metrics.specs	Per-metric specifications used by JEDI and Φ.
rpx_benchmark.metrics.tracking	Multi-object tracking metric calculators.
rpx_benchmark.metrics.tracking_paper	Paper-protocol metrics for RPX D3 multi-object tracking.
rpx_benchmark.metrics.video_depth	Metric calculators bound to TaskType.VIDEO_DEPTH (paper task Video Depth).

Task runners

Module	Purpose
rpx_benchmark.tasks	Task runners + registry.
rpx_benchmark.tasks._pipeline	Shared runner implementation; use the task-specific RunConfig and run_* interfaces.
rpx_benchmark.tasks._video_pipeline	Shared runner implementation; use the task-specific RunConfig and run_* interfaces.
rpx_benchmark.tasks.detection	Object detection (closed vocab + open vocab).
rpx_benchmark.tasks.keypoint_matching	Keypoint matching between a pair of RGB frames.
rpx_benchmark.tasks.monocular_depth	Monocular absolute depth.
rpx_benchmark.tasks.registry	Task plugin registry.
rpx_benchmark.tasks.relative_pose	Relative camera pose between two frames.
rpx_benchmark.tasks.robot_vqa	Robot-grounded VQA task definitions.
rpx_benchmark.tasks.robot_vqa_v2	Robot VQA v2: three context modes, with all ground truth from recorded data.
rpx_benchmark.tasks.segmentation	Object (instance) segmentation.
rpx_benchmark.tasks.sparse_depth	Sparse depth: predict depth at a set of pixel locations.
rpx_benchmark.tasks.tracking	Object tracking: per-frame MOT with stable track IDs.
rpx_benchmark.tasks.video_depth	Video absolute depth (paper task Video Depth).
rpx_benchmark.tasks.visual_grounding	Visual grounding: referring expression -> box.
rpx_benchmark.tasks.vqa_generation	RPX VQA ground-truth generation.

VQA contracts and evaluation

Module	Purpose
rpx_benchmark.vqa	Benchmark contracts for RPX's single-image VQA tasks.
rpx_benchmark.vqa.canonical	Canonical forms used to avoid sampling semantically duplicate questions.
rpx_benchmark.vqa.contract	Canonical, model-independent contract for RPX VQA benchmark rows.
rpx_benchmark.vqa.evaluate	Run any callable VQA model with the published RPX prompts and scoring.
rpx_benchmark.vqa.hub_rgb	Fetches single images from RPX's uncompressed RGB tar shards over HTTP range requests.
rpx_benchmark.vqa.metrics	Benchmark metrics and smoke-gate aggregation for RPX VQA.
rpx_benchmark.vqa.outputs	Strict parsing and normalization of VQA model outputs.
rpx_benchmark.vqa.prompts	Task prompts before each model's official chat/image template is applied.
rpx_benchmark.vqa.roster	Frozen single-image model roster supplied for the RPX VQA benchmark.
rpx_benchmark.vqa.sampling	Deterministic row selection that prefers diverse samples.

The reference pages are searchable. The guides have runnable examples; the module pages have the full contracts.

Installed workflow examples

Module	Purpose
examples.benchmark_tasks	Smoke fixtures for the six tasks and the benchmark_tasks command line.
examples.profile_callable	Times a callable with device synchronization and records memory and a system card.
examples.summarize_metrics	Φ and JEDI summaries from raw CSV, JSON or JSONL metric records.