Toolkit capabilities¶
Everything the toolkit can do, and where it lives. The benchmark runners and analysis APIs ship in the PyPI package; annotation tools, the paper's model runners and their containers are in the repository.
| Capability | Where to start | Where it lives |
|---|---|---|
| Discover, download and load data | Data contracts, Hub API | PyPI; Hub extra for downloads/Parquet |
| Validate manifests and units | Schemas API | PyPI schemas extra |
| Six paper task interfaces | Task benchmark examples | PyPI; pretrained dependencies separate |
| Use local or API models | Bring your own model | PyPI adapters and VQA callable contract |
| Annotate and refine GT masks | Mask pipeline | Source checkout; CUDA and review UI |
| Measure latency and memory | Hardware profiler | PyPI; PyTorch/CUDA for GPU measurement |
| Compute phase robustness and desirability | Φ/JEDI calculator | PyPI; raw paired metrics required |
| Register new metrics or runners | Add metrics, Add tasks | PyPI runtime registration; new identities require source changes |
| Score scene difficulty | ESD extraction, ESD scoring | PyPI; analysis/optional decoding extras |
| Build manifests and tar shards | Dataset hub CLI | PyPI Hub tools |
| Inspect scenes interactively | Rerun scene tool | Repository script; viz extra |
| Detection, segmentation, sparse depth, keypoint matching | Task API inventory | Additional registered PyPI task runners |
| Canonical question generation and VQA validation | VQA source, VQA APIs | Generation in source; evaluation in PyPI |
| Per-sample/cell logs and structured reports | Cell log API, Reports API | PyPI |
| Batch adapters and depth alignment | Adapter inventory, Alignment API | PyPI; model support/protocol dependent |
Discover the installed registries¶
from rpx_benchmark.tasks import available_tasks
from rpx_benchmark.metrics import available_metrics
print([task.value for task in available_tasks()])
print(available_metrics())
Registered tasks are reusable APIs; the paper's six tasks each add a frozen evaluation protocol on top. A new model function or metric doesn't change those protocols.