rpx_benchmark.api
Core task enums, data contracts, and the base model interface.
This module is the stable public surface users bind against when they
plug a model into the benchmark. Everything here is either an
Enum, a plain @dataclass prediction/ground-truth container, or
the BenchmarkModel abstract base that defines what a model
looks like to the runner.
The three pluggable systems that sit on top of these types are:
rpx_benchmark.adapters— turns an arbitrary model into aBenchmarkModel-shaped object via theInputAdapter / model / OutputAdaptercontract.rpx_benchmark.metrics— task → calculator plugin registry.rpx_benchmark.tasks.registry— task → runner plugin registry.
Stability
Enums and dataclasses in this module are append-only: adding new tasks or new fields is fine; renaming or removing them is a breaking change that requires a major version bump.
1"""Core task enums, data contracts, and the base model interface. 2 3This module is the stable public surface users bind against when they 4plug a model into the benchmark. Everything here is either an 5``Enum``, a plain ``@dataclass`` prediction/ground-truth container, or 6the :class:`BenchmarkModel` abstract base that defines what a model 7looks like to the runner. 8 9The three pluggable systems that sit on top of these types are: 10 11- :mod:`rpx_benchmark.adapters` — turns an arbitrary model into a 12 :class:`BenchmarkModel`-shaped object via the 13 ``InputAdapter / model / OutputAdapter`` contract. 14- :mod:`rpx_benchmark.metrics` — task → calculator plugin registry. 15- :mod:`rpx_benchmark.tasks.registry` — task → runner plugin registry. 16 17Stability 18--------- 19 20Enums and dataclasses in this module are append-only: adding new 21tasks or new fields is fine; renaming or removing them is a breaking 22change that requires a major version bump. 23""" 24 25from __future__ import annotations 26 27from abc import ABC, abstractmethod 28from dataclasses import dataclass 29from enum import Enum 30from typing import Any, Dict, Literal, Sequence 31 32import numpy as np 33 34 35class TaskType(str, Enum): 36 """Enumeration of every task the benchmark toolkit recognises. 37 38 Members are plain strings so they serialise cleanly to JSON and 39 can be used as dict keys for logging / table rows. 40 41 Members 42 ------- 43 MONOCULAR_DEPTH 44 Dense metric depth from a single RGB frame (paper task Image Depth). 45 VIDEO_DEPTH 46 Dense metric depth from a full phase clip (~250 RGB frames), 47 producing a per-frame depth sequence with temporal consistency 48 (paper task Video Depth). Same scenes and ground truth as 49 MONOCULAR_DEPTH; differs in input context and metric set 50 (per-frame metrics + temporal metrics). 51 OBJECT_DETECTION 52 Closed-vocabulary detection with category labels. 53 OBJECT_SEGMENTATION 54 Instance segmentation masks with per-pixel instance IDs. 55 OBJECT_TRACKING 56 Multi-object tracking with persistent track IDs. 57 RELATIVE_CAMERA_POSE 58 6-DoF pose of frame B relative to frame A. 59 OPEN_VOCAB_DETECTION 60 Detection conditioned on a free-text vocabulary. 61 VISUAL_GROUNDING 62 Referring expression → bounding box on the image. 63 SPARSE_DEPTH 64 Depth values at a sparse set of image locations only. 65 KEYPOINT_MATCHING 66 Dense/sparse correspondences between two images. 67 68 Examples 69 -------- 70 >>> from rpx_benchmark.api import TaskType 71 >>> TaskType.MONOCULAR_DEPTH.value 72 'monocular_depth' 73 >>> TaskType("monocular_depth") is TaskType.MONOCULAR_DEPTH 74 True 75 """ 76 77 MONOCULAR_DEPTH = "monocular_depth" 78 VIDEO_DEPTH = "video_depth" 79 OBJECT_DETECTION = "object_detection" 80 OBJECT_SEGMENTATION = "object_segmentation" 81 OBJECT_TRACKING = "object_tracking" 82 RELATIVE_CAMERA_POSE = "relative_camera_pose" 83 OPEN_VOCAB_DETECTION = "open_vocab_detection" 84 VISUAL_GROUNDING = "visual_grounding" 85 SPARSE_DEPTH = "sparse_depth" 86 KEYPOINT_MATCHING = "keypoint_matching" 87 88 89class Phase(str, Enum): 90 """Capture phases of the three-phase RPX reconfiguration protocol. 91 92 Every scene is recorded in three phases so the benchmark can 93 attribute performance changes to scene state rather than to 94 lighting / viewpoint / camera identity. 95 96 Members 97 ------- 98 CLUTTER 99 Initial dense object arrangement; significant inter-object 100 occlusion. 101 INTERACTION 102 Human operator grasps and moves objects. Introduces 103 hand-object contact and transient occlusion. 104 CLEAN 105 Same objects re-organised sparsely. Serves as a within-scene 106 control for the other two phases. 107 """ 108 109 CLUTTER = "clutter" 110 INTERACTION = "interaction" 111 CLEAN = "clean" 112 113 114class Difficulty(str, Enum): 115 """Effort-Stratified Difficulty (ESD) split label. 116 117 ESD splits are derived per ``(scene, phase)`` from the 118 annotation-effort signal described in paper §4. See 119 :mod:`rpx_benchmark.deployment` for the scoring details. 120 121 Members 122 ------- 123 EASY 124 Few annotation iterations, low occlusion, stable visibility. 125 MEDIUM 126 HARD 127 Many annotation iterations, dense occlusion, high depth- 128 invalid fraction, high jerk. 129 """ 130 131 EASY = "easy" 132 MEDIUM = "medium" 133 HARD = "hard" 134 135 136#: Weights used for the ESD-weighted phase score 137#: :math:`S_p = 0.25 E + 0.35 M + 0.40 H`. 138ESD_WEIGHTS: Dict[Difficulty, float] = { 139 Difficulty.EASY: 0.25, 140 Difficulty.MEDIUM: 0.35, 141 Difficulty.HARD: 0.40, 142} 143 144 145@dataclass 146class DepthGroundTruth: 147 depth_map: np.ndarray # H x W float32 meters 148 149 150@dataclass 151class VideoDepthGroundTruth: 152 """Per-clip ground truth for the Video Depth task. 153 154 The model receives the full phase clip as an RGB sequence and is 155 scored against a depth sequence of the same shape. ``valid_mask_seq`` 156 follows the same D435 sentinel convention as 157 :class:`DepthGroundTruth` (zero entries mark invalid pixels — no 158 IR return); the loader populates it once when reading the clip 159 so per-frame masking is cheap during metric computation. 160 161 Shapes 162 ------ 163 depth_map_seq : (T, H, W) float32, metres 164 valid_mask_seq : (T, H, W) bool, finite GT with 0.3 < depth < 5.0 m 165 frame_indices : (T,) int32, original phase-relative frame indices 166 so models that want temporal context can use them 167 compute_fscore : bool, opt-in diagnostic switch; false for headline D1-V 168 """ 169 170 depth_map_seq: np.ndarray 171 valid_mask_seq: np.ndarray 172 frame_indices: np.ndarray 173 compute_fscore: bool = False 174 175 176@dataclass 177class DetectionGroundTruth: 178 boxes: np.ndarray # N x 4 float (x1, y1, x2, y2) 179 labels: Sequence[str] # class names aligned with boxes 180 181 182@dataclass 183class SegmentationGroundTruth: 184 mask: np.ndarray # H x W int (class ids) 185 186 187@dataclass 188class Tracklet: 189 track_id: str 190 boxes: np.ndarray # T x 4 float (x1, y1, x2, y2) 191 scores: np.ndarray | None = None # Optional T float 192 193 194@dataclass 195class TrackletGroundTruth: 196 tracks: Sequence[Tracklet] 197 198 199@dataclass 200class VisualGroundingGroundTruth: 201 text: str 202 boxes: np.ndarray # N x 4 float 203 labels: Sequence[str] | None = None 204 205 206@dataclass 207class RelativePoseGroundTruth: 208 rotation: np.ndarray # 3 x 3 or 4-array quaternion 209 translation: np.ndarray # 3-vector meters 210 211 212@dataclass 213class SparseDepthGroundTruth: 214 coordinates: np.ndarray # N x 2 (x, y) pixel coords 215 depths: np.ndarray # N float depths (m) 216 217 218 219 220@dataclass 221class KeypointCorrespondenceGroundTruth: 222 points0: np.ndarray # N x 2 float 223 points1: np.ndarray # N x 2 float 224 visibility: np.ndarray | None = None # optional mask 225 226 227@dataclass 228class Sample: 229 """One input unit handed by :class:`RPXDataset` to a model. 230 231 Samples are produced by the loader and consumed by 232 ``BenchmarkModel.predict``. Every field is deliberately simple 233 (numpy arrays, enums, plain dicts) so models and adapters don't 234 need to know anything about the on-disk dataset format. 235 236 Parameters 237 ---------- 238 id : str 239 Unique identifier of the form ``{scene}_{phase}_{frame}``. 240 Used for joining per-sample metrics back to scenes / phases. 241 rgb : np.ndarray 242 H × W × 3 uint8 RGB image in row-major order. 243 ground_truth : Any 244 Task-specific GroundTruth dataclass (e.g. 245 :class:`DepthGroundTruth`). The concrete type is determined 246 by :attr:`RPXDataset.task`. 247 metadata : dict, optional 248 Free-form metadata the loader can attach; conventionally 249 holds fisheye images, secondary RGB frames for pair tasks, 250 and any label paths that do not fit into the ground-truth 251 dataclass. Consumers should treat unknown keys as opaque. 252 phase : Phase, optional 253 Capture phase the frame belongs to. Required for ESD-weighted 254 phase scoring. 255 difficulty : Difficulty, optional 256 ESD difficulty label of the ``(scene, phase)`` this sample 257 belongs to. 258 camera_pose : np.ndarray, optional 259 4 × 4 float64 SE(3) matrix (camera → world) sourced from the 260 T265 tracker. Used for the temporal-stability metric. 261 """ 262 263 id: str 264 rgb: np.ndarray 265 ground_truth: Any 266 metadata: Dict[str, Any] | None = None 267 phase: Phase | None = None 268 difficulty: Difficulty | None = None 269 camera_pose: np.ndarray | None = None 270 271 272@dataclass 273class VideoSample: 274 """One input unit for video tasks (Video Depth today, possibly more later). 275 276 Mirrors :class:`Sample` but carries sequence-shaped fields so a 277 video model can ingest the entire phase clip in one call. The 278 iteration unit is a ``(scene, phase)`` clip, not a frame, so the 279 cell-log key for video tasks is ``(model, task, scene, phase)`` 280 rather than ``(..., frame)``. 281 282 Shapes 283 ------ 284 id : ``"{scene}_{phase}"`` 285 rgb_seq : (T, H, W, 3) uint8 286 ground_truth : task-specific (e.g. :class:`VideoDepthGroundTruth`) 287 metadata : dict — must include ``frame_indices`` (T,) int32, 288 ``intrinsics`` (3, 3) float32, and optionally 289 ``ego_motion_seq`` (T, 4, 4) float64 from T265 290 phase : :class:`Phase` enum value for this clip 291 difficulty : :class:`Difficulty` of the ``(scene, phase)`` 292 camera_pose_seq : (T, 4, 4) float64, camera → world per frame, or 293 ``None`` if the task does not need pose context 294 295 The split from :class:`Sample` is deliberate: trying to bolt 296 sequence fields into a single Sample type would have made 297 every per-frame consumer guard against ``rgb_seq is None``. Two 298 explicit shapes keep adapters honest about which task they implement. 299 """ 300 301 id: str 302 rgb_seq: np.ndarray 303 ground_truth: Any 304 metadata: Dict[str, Any] | None = None 305 phase: Phase | None = None 306 difficulty: Difficulty | None = None 307 camera_pose_seq: np.ndarray | None = None 308 309 310@dataclass 311class DepthPrediction: 312 depth_map: np.ndarray # H x W float32 313 314 315@dataclass 316class VideoDepthPrediction: 317 """A video depth model's output for one phase clip. 318 319 Shape 320 ----- 321 depth_map_seq : (T, H, W) float32, metres. Models that emit 322 affine-invariant (relative) depth should still return a 323 metric-scale tensor; the loader/runner will handle per-clip 324 scale-and-shift alignment before metric computation (canonical 325 Ranftl et al. 2020 procedure). Aligning per-frame would defeat 326 the temporal consistency metrics; aligning per-clip is the 327 standard convention for video-depth evaluation. 328 """ 329 330 depth_map_seq: np.ndarray 331 332 333@dataclass 334class DetectionPrediction: 335 boxes: np.ndarray # N x 4 float 336 scores: np.ndarray # N float 337 labels: Sequence[str] 338 339 340@dataclass 341class SegmentationPrediction: 342 mask: np.ndarray # H x W int (class ids) 343 344 345@dataclass 346class TrackletPrediction: 347 tracks: Sequence[Tracklet] 348 349 350@dataclass 351class VisualGroundingPrediction: 352 boxes: np.ndarray # N x 4 float 353 scores: np.ndarray # N float 354 labels: Sequence[str] | None = None 355 356 357@dataclass 358class RelativePosePrediction: 359 rotation: np.ndarray # 3 x 3 or quaternion 360 translation: np.ndarray # 3-vector meters 361 362 363@dataclass 364class SparseDepthPrediction: 365 coordinates: np.ndarray # N x 2 366 depths: np.ndarray # N 367 368 369 370 371@dataclass 372class KeypointCorrespondencePrediction: 373 points0: np.ndarray # N x 2 374 points1: np.ndarray # N x 2 375 scores: np.ndarray | None = None 376 377 378class BenchmarkModel(ABC): 379 """Abstract base class every RPX-compatible model must implement. 380 381 In practice, most users should **not** subclass this directly — 382 instead compose a :class:`rpx_benchmark.adapters.BenchmarkableModel` 383 from an input adapter, a model callable, and an output adapter. 384 ``BenchmarkableModel`` already implements :meth:`predict` and 385 :meth:`setup` correctly for you. 386 387 Subclass only when you need complete control over how samples 388 are routed to your model (e.g. true minibatching across GPU 389 devices). 390 391 Attributes 392 ---------- 393 task : TaskType 394 The task this model solves. Must be set by subclasses (either 395 at class level or in ``__init__``). The runner checks that 396 ``model.task == dataset.task`` before running. 397 depth_output_kind : Literal["metric", "relative"] 398 Only meaningful for depth tasks (``MONOCULAR_DEPTH`` and 399 ``VIDEO_DEPTH``). Declares whether the model emits depth in 400 physical units (``"metric"``, default — passes through the 401 runner unchanged) or up to an unknown scale and shift 402 (``"relative"`` — runner applies per-scene-phase scale+shift 403 alignment via :func:`~rpx_benchmark.metrics.depth_alignment.align_pred_to_gt_pooled` 404 before computing metrics, matching the Ranftl et al. 2020 405 protocol the paper §3.3 prescribes). 406 407 Adapters that wrap relative-depth models (Lotus-2, FE2E and 408 the DA-V2 relative variant) must set 409 this to ``"relative"``. Adapters wrapping metric models 410 (DA-Metric, Depth Pro, UniDepth V2, Metric3D V2, MoGe-2, 411 canonical HyDen metric, DepthLM, etc.) leave 412 the default. Ignored entirely for non-depth tasks. 413 414 Examples 415 -------- 416 Minimal subclass:: 417 418 class MyDepth(BenchmarkModel): 419 task = TaskType.MONOCULAR_DEPTH 420 # depth_output_kind defaults to "metric" 421 422 def setup(self): 423 self.net = load_my_checkpoint() 424 425 def predict(self, batch): 426 return [ 427 DepthPrediction(depth_map=self.net(s.rgb)) 428 for s in batch 429 ] 430 431 Relative-depth wrapper (Lotus-2, FE2E, MoGe-2, ...):: 432 433 class MyRelativeDepth(BenchmarkModel): 434 task = TaskType.MONOCULAR_DEPTH 435 depth_output_kind = "relative" 436 437 def setup(self): ... 438 def predict(self, batch): ... # returns up-to-scale depth 439 440 Composed via :class:`BenchmarkableModel`:: 441 442 bm = rpx.BenchmarkableModel( 443 task=TaskType.MONOCULAR_DEPTH, 444 input_adapter=MyInputAdapter(), 445 model=my_nn_module, 446 output_adapter=MyOutputAdapter(), 447 name="my_model", 448 ) 449 """ 450 451 task: TaskType 452 #: Depth output convention — ``"metric"`` (default) means the 453 #: prediction is in metres and goes straight to the metric 454 #: calculators. ``"relative"`` means the prediction is up to an 455 #: unknown scale + shift; the runner applies per-scene-phase 456 #: pooled scale-and-shift alignment via 457 #: :func:`~rpx_benchmark.metrics.depth_alignment.align_pred_to_gt_pooled` 458 #: before computing metrics. Ignored for non-depth tasks. 459 depth_output_kind: Literal["metric", "relative"] = "metric" 460 461 @abstractmethod 462 def setup(self) -> None: 463 """Load checkpoints, warm CUDA, and do any other one-time init. 464 465 The runner calls this exactly once before iterating the 466 dataset, unless ``BenchmarkRunner(call_setup=False)`` was 467 passed — in which case the caller is responsible. 468 """ 469 470 @abstractmethod 471 def predict(self, batch: Sequence[Sample]) -> Sequence[Any]: 472 """Run inference on a batch of samples. 473 474 Parameters 475 ---------- 476 batch : sequence of Sample 477 One or more samples. Length equals ``dataset.batch_size`` 478 except possibly for the final tail batch. 479 480 Returns 481 ------- 482 sequence 483 One task-specific Prediction dataclass per input sample, 484 in the same order. The prediction dataclass must match 485 what :class:`MetricSuite` expects for this task. 486 487 Raises 488 ------ 489 ModelError 490 (By convention) when a sample cannot be processed. The 491 runner surfaces it as a clean error rather than a stack 492 trace. 493 """ 494 495 496def _check_shape(arr: np.ndarray, dims: int, name: str) -> None: 497 """Raise :class:`ModelError` if ``arr`` does not have exactly ``dims`` axes.""" 498 if arr.ndim != dims: 499 from .exceptions import ModelError 500 501 raise ModelError( 502 f"{name} must have {dims} dims; got {arr.ndim}", 503 hint=f"Check the shape your model returns for {name!r}.", 504 ) 505 506 507def validate_prediction(task: TaskType, prediction: Any, sample: Sample | None = None) -> None: 508 """Validate a Prediction dataclass's shape and type for a given task. 509 510 Parameters 511 ---------- 512 task : TaskType 513 Task the runner is evaluating. 514 prediction : Any 515 Prediction dataclass the model just returned. 516 sample : Sample, optional 517 The sample the prediction was produced for; used for shape 518 cross-checks (e.g. segmentation mask vs RGB size). 519 520 Raises 521 ------ 522 ModelError 523 If the prediction is the wrong type or the wrong shape for the 524 task. 525 """ 526 from .exceptions import ModelError 527 528 def _type_error(expected: str) -> "ModelError": 529 return ModelError( 530 f"{task.value} models must return {expected}, got {type(prediction).__name__}", 531 hint=f"Check your model's predict() return type for {task.value}.", 532 ) 533 534 if task == TaskType.MONOCULAR_DEPTH: 535 if not isinstance(prediction, DepthPrediction): 536 raise _type_error("DepthPrediction") 537 _check_shape(prediction.depth_map, 2, "Depth map") 538 return 539 540 if task in (TaskType.OBJECT_DETECTION, TaskType.OPEN_VOCAB_DETECTION): 541 if not isinstance(prediction, DetectionPrediction): 542 raise _type_error("DetectionPrediction") 543 boxes = prediction.boxes 544 _check_shape(boxes, 2, "Detection boxes") 545 if boxes.shape[1] != 4: 546 raise ModelError( 547 f"Detection boxes must have shape [N, 4]; got {boxes.shape}", 548 ) 549 if len(prediction.scores) != len(boxes) or len(prediction.labels) != len(boxes): 550 raise ModelError( 551 "DetectionPrediction boxes, scores, and labels must all have the same length.", 552 details={ 553 "n_boxes": len(boxes), 554 "n_scores": len(prediction.scores), 555 "n_labels": len(prediction.labels), 556 }, 557 ) 558 return 559 560 if task == TaskType.OBJECT_SEGMENTATION: 561 if not isinstance(prediction, SegmentationPrediction): 562 raise _type_error("SegmentationPrediction") 563 _check_shape(prediction.mask, 2, "Segmentation mask") 564 if sample and prediction.mask.shape != sample.rgb.shape[:2]: 565 raise ModelError( 566 f"Segmentation mask shape {prediction.mask.shape} must " 567 f"match the RGB spatial size {sample.rgb.shape[:2]}", 568 hint="The OutputAdapter should resize masks to the sample's " 569 "original H x W before returning.", 570 ) 571 return 572 573 if task == TaskType.OBJECT_TRACKING: 574 if not isinstance(prediction, TrackletPrediction): 575 raise _type_error("TrackletPrediction") 576 return 577 578 if task == TaskType.VISUAL_GROUNDING: 579 if not isinstance(prediction, VisualGroundingPrediction): 580 raise _type_error("VisualGroundingPrediction") 581 return 582 583 if task == TaskType.RELATIVE_CAMERA_POSE: 584 if not isinstance(prediction, RelativePosePrediction): 585 raise _type_error("RelativePosePrediction") 586 return 587 588 if task == TaskType.SPARSE_DEPTH: 589 if not isinstance(prediction, SparseDepthPrediction): 590 raise _type_error("SparseDepthPrediction") 591 return 592 593 594 if task == TaskType.KEYPOINT_MATCHING: 595 if not isinstance(prediction, KeypointCorrespondencePrediction): 596 raise _type_error("KeypointCorrespondencePrediction") 597 return 598 599 raise ModelError(f"Unsupported task: {task}")
class
TaskType(builtins.str, enum.Enum):
36class TaskType(str, Enum): 37 """Enumeration of every task the benchmark toolkit recognises. 38 39 Members are plain strings so they serialise cleanly to JSON and 40 can be used as dict keys for logging / table rows. 41 42 Members 43 ------- 44 MONOCULAR_DEPTH 45 Dense metric depth from a single RGB frame (paper task Image Depth). 46 VIDEO_DEPTH 47 Dense metric depth from a full phase clip (~250 RGB frames), 48 producing a per-frame depth sequence with temporal consistency 49 (paper task Video Depth). Same scenes and ground truth as 50 MONOCULAR_DEPTH; differs in input context and metric set 51 (per-frame metrics + temporal metrics). 52 OBJECT_DETECTION 53 Closed-vocabulary detection with category labels. 54 OBJECT_SEGMENTATION 55 Instance segmentation masks with per-pixel instance IDs. 56 OBJECT_TRACKING 57 Multi-object tracking with persistent track IDs. 58 RELATIVE_CAMERA_POSE 59 6-DoF pose of frame B relative to frame A. 60 OPEN_VOCAB_DETECTION 61 Detection conditioned on a free-text vocabulary. 62 VISUAL_GROUNDING 63 Referring expression → bounding box on the image. 64 SPARSE_DEPTH 65 Depth values at a sparse set of image locations only. 66 KEYPOINT_MATCHING 67 Dense/sparse correspondences between two images. 68 69 Examples 70 -------- 71 >>> from rpx_benchmark.api import TaskType 72 >>> TaskType.MONOCULAR_DEPTH.value 73 'monocular_depth' 74 >>> TaskType("monocular_depth") is TaskType.MONOCULAR_DEPTH 75 True 76 """ 77 78 MONOCULAR_DEPTH = "monocular_depth" 79 VIDEO_DEPTH = "video_depth" 80 OBJECT_DETECTION = "object_detection" 81 OBJECT_SEGMENTATION = "object_segmentation" 82 OBJECT_TRACKING = "object_tracking" 83 RELATIVE_CAMERA_POSE = "relative_camera_pose" 84 OPEN_VOCAB_DETECTION = "open_vocab_detection" 85 VISUAL_GROUNDING = "visual_grounding" 86 SPARSE_DEPTH = "sparse_depth" 87 KEYPOINT_MATCHING = "keypoint_matching"
Enumeration of every task the benchmark toolkit recognises.
Members are plain strings so they serialise cleanly to JSON and can be used as dict keys for logging / table rows.
Members
MONOCULAR_DEPTH Dense metric depth from a single RGB frame (paper task Image Depth). VIDEO_DEPTH Dense metric depth from a full phase clip (~250 RGB frames), producing a per-frame depth sequence with temporal consistency (paper task Video Depth). Same scenes and ground truth as MONOCULAR_DEPTH; differs in input context and metric set (per-frame metrics + temporal metrics). OBJECT_DETECTION Closed-vocabulary detection with category labels. OBJECT_SEGMENTATION Instance segmentation masks with per-pixel instance IDs. OBJECT_TRACKING Multi-object tracking with persistent track IDs. RELATIVE_CAMERA_POSE 6-DoF pose of frame B relative to frame A. OPEN_VOCAB_DETECTION Detection conditioned on a free-text vocabulary. VISUAL_GROUNDING Referring expression → bounding box on the image. SPARSE_DEPTH Depth values at a sparse set of image locations only. KEYPOINT_MATCHING Dense/sparse correspondences between two images.
Examples
>>> from rpx_benchmark.api import TaskType
>>> TaskType.MONOCULAR_DEPTH.value
'monocular_depth'
>>> TaskType("monocular_depth") is TaskType.MONOCULAR_DEPTH
True
MONOCULAR_DEPTH =
<TaskType.MONOCULAR_DEPTH: 'monocular_depth'>
VIDEO_DEPTH =
<TaskType.VIDEO_DEPTH: 'video_depth'>
OBJECT_DETECTION =
<TaskType.OBJECT_DETECTION: 'object_detection'>
OBJECT_SEGMENTATION =
<TaskType.OBJECT_SEGMENTATION: 'object_segmentation'>
OBJECT_TRACKING =
<TaskType.OBJECT_TRACKING: 'object_tracking'>
RELATIVE_CAMERA_POSE =
<TaskType.RELATIVE_CAMERA_POSE: 'relative_camera_pose'>
OPEN_VOCAB_DETECTION =
<TaskType.OPEN_VOCAB_DETECTION: 'open_vocab_detection'>
VISUAL_GROUNDING =
<TaskType.VISUAL_GROUNDING: 'visual_grounding'>
SPARSE_DEPTH =
<TaskType.SPARSE_DEPTH: 'sparse_depth'>
KEYPOINT_MATCHING =
<TaskType.KEYPOINT_MATCHING: 'keypoint_matching'>
class
Phase(builtins.str, enum.Enum):
90class Phase(str, Enum): 91 """Capture phases of the three-phase RPX reconfiguration protocol. 92 93 Every scene is recorded in three phases so the benchmark can 94 attribute performance changes to scene state rather than to 95 lighting / viewpoint / camera identity. 96 97 Members 98 ------- 99 CLUTTER 100 Initial dense object arrangement; significant inter-object 101 occlusion. 102 INTERACTION 103 Human operator grasps and moves objects. Introduces 104 hand-object contact and transient occlusion. 105 CLEAN 106 Same objects re-organised sparsely. Serves as a within-scene 107 control for the other two phases. 108 """ 109 110 CLUTTER = "clutter" 111 INTERACTION = "interaction" 112 CLEAN = "clean"
Capture phases of the three-phase RPX reconfiguration protocol.
Every scene is recorded in three phases so the benchmark can attribute performance changes to scene state rather than to lighting / viewpoint / camera identity.
Members
CLUTTER Initial dense object arrangement; significant inter-object occlusion. INTERACTION Human operator grasps and moves objects. Introduces hand-object contact and transient occlusion. CLEAN Same objects re-organised sparsely. Serves as a within-scene control for the other two phases.
CLUTTER =
<Phase.CLUTTER: 'clutter'>
INTERACTION =
<Phase.INTERACTION: 'interaction'>
CLEAN =
<Phase.CLEAN: 'clean'>
class
Difficulty(builtins.str, enum.Enum):
115class Difficulty(str, Enum): 116 """Effort-Stratified Difficulty (ESD) split label. 117 118 ESD splits are derived per ``(scene, phase)`` from the 119 annotation-effort signal described in paper §4. See 120 :mod:`rpx_benchmark.deployment` for the scoring details. 121 122 Members 123 ------- 124 EASY 125 Few annotation iterations, low occlusion, stable visibility. 126 MEDIUM 127 HARD 128 Many annotation iterations, dense occlusion, high depth- 129 invalid fraction, high jerk. 130 """ 131 132 EASY = "easy" 133 MEDIUM = "medium" 134 HARD = "hard"
Effort-Stratified Difficulty (ESD) split label.
ESD splits are derived per (scene, phase) from the
annotation-effort signal described in paper §4. See
rpx_benchmark.deployment for the scoring details.
Members
EASY Few annotation iterations, low occlusion, stable visibility. MEDIUM HARD Many annotation iterations, dense occlusion, high depth- invalid fraction, high jerk.
EASY =
<Difficulty.EASY: 'easy'>
MEDIUM =
<Difficulty.MEDIUM: 'medium'>
HARD =
<Difficulty.HARD: 'hard'>
ESD_WEIGHTS: Dict[Difficulty, float] =
{<Difficulty.EASY: 'easy'>: 0.25, <Difficulty.MEDIUM: 'medium'>: 0.35, <Difficulty.HARD: 'hard'>: 0.4}
@dataclass
class
DepthGroundTruth:
@dataclass
class
VideoDepthGroundTruth:
151@dataclass 152class VideoDepthGroundTruth: 153 """Per-clip ground truth for the Video Depth task. 154 155 The model receives the full phase clip as an RGB sequence and is 156 scored against a depth sequence of the same shape. ``valid_mask_seq`` 157 follows the same D435 sentinel convention as 158 :class:`DepthGroundTruth` (zero entries mark invalid pixels — no 159 IR return); the loader populates it once when reading the clip 160 so per-frame masking is cheap during metric computation. 161 162 Shapes 163 ------ 164 depth_map_seq : (T, H, W) float32, metres 165 valid_mask_seq : (T, H, W) bool, finite GT with 0.3 < depth < 5.0 m 166 frame_indices : (T,) int32, original phase-relative frame indices 167 so models that want temporal context can use them 168 compute_fscore : bool, opt-in diagnostic switch; false for headline D1-V 169 """ 170 171 depth_map_seq: np.ndarray 172 valid_mask_seq: np.ndarray 173 frame_indices: np.ndarray 174 compute_fscore: bool = False
Per-clip ground truth for the Video Depth task.
The model receives the full phase clip as an RGB sequence and is
scored against a depth sequence of the same shape. valid_mask_seq
follows the same D435 sentinel convention as
DepthGroundTruth (zero entries mark invalid pixels — no
IR return); the loader populates it once when reading the clip
so per-frame masking is cheap during metric computation.
Shapes
depth_map_seq : (T, H, W) float32, metres valid_mask_seq : (T, H, W) bool, finite GT with 0.3 < depth < 5.0 m frame_indices : (T,) int32, original phase-relative frame indices so models that want temporal context can use them compute_fscore : bool, opt-in diagnostic switch; false for headline D1-V
@dataclass
class
DetectionGroundTruth:
@dataclass
class
SegmentationGroundTruth:
@dataclass
class
Tracklet:
@dataclass
class
TrackletGroundTruth:
TrackletGroundTruth(tracks: Sequence[Tracklet])
tracks: Sequence[Tracklet]
@dataclass
class
VisualGroundingGroundTruth:
200@dataclass 201class VisualGroundingGroundTruth: 202 text: str 203 boxes: np.ndarray # N x 4 float 204 labels: Sequence[str] | None = None
@dataclass
class
RelativePoseGroundTruth:
@dataclass
class
SparseDepthGroundTruth:
@dataclass
class
KeypointCorrespondenceGroundTruth:
221@dataclass 222class KeypointCorrespondenceGroundTruth: 223 points0: np.ndarray # N x 2 float 224 points1: np.ndarray # N x 2 float 225 visibility: np.ndarray | None = None # optional mask
@dataclass
class
Sample:
228@dataclass 229class Sample: 230 """One input unit handed by :class:`RPXDataset` to a model. 231 232 Samples are produced by the loader and consumed by 233 ``BenchmarkModel.predict``. Every field is deliberately simple 234 (numpy arrays, enums, plain dicts) so models and adapters don't 235 need to know anything about the on-disk dataset format. 236 237 Parameters 238 ---------- 239 id : str 240 Unique identifier of the form ``{scene}_{phase}_{frame}``. 241 Used for joining per-sample metrics back to scenes / phases. 242 rgb : np.ndarray 243 H × W × 3 uint8 RGB image in row-major order. 244 ground_truth : Any 245 Task-specific GroundTruth dataclass (e.g. 246 :class:`DepthGroundTruth`). The concrete type is determined 247 by :attr:`RPXDataset.task`. 248 metadata : dict, optional 249 Free-form metadata the loader can attach; conventionally 250 holds fisheye images, secondary RGB frames for pair tasks, 251 and any label paths that do not fit into the ground-truth 252 dataclass. Consumers should treat unknown keys as opaque. 253 phase : Phase, optional 254 Capture phase the frame belongs to. Required for ESD-weighted 255 phase scoring. 256 difficulty : Difficulty, optional 257 ESD difficulty label of the ``(scene, phase)`` this sample 258 belongs to. 259 camera_pose : np.ndarray, optional 260 4 × 4 float64 SE(3) matrix (camera → world) sourced from the 261 T265 tracker. Used for the temporal-stability metric. 262 """ 263 264 id: str 265 rgb: np.ndarray 266 ground_truth: Any 267 metadata: Dict[str, Any] | None = None 268 phase: Phase | None = None 269 difficulty: Difficulty | None = None 270 camera_pose: np.ndarray | None = None
One input unit handed by RPXDataset to a model.
Samples are produced by the loader and consumed by
BenchmarkModel.predict. Every field is deliberately simple
(numpy arrays, enums, plain dicts) so models and adapters don't
need to know anything about the on-disk dataset format.
Parameters
id : str
Unique identifier of the form {scene}_{phase}_{frame}.
Used for joining per-sample metrics back to scenes / phases.
rgb : np.ndarray
H × W × 3 uint8 RGB image in row-major order.
ground_truth : Any
Task-specific GroundTruth dataclass (e.g.
DepthGroundTruth). The concrete type is determined
by RPXDataset.task.
metadata : dict, optional
Free-form metadata the loader can attach; conventionally
holds fisheye images, secondary RGB frames for pair tasks,
and any label paths that do not fit into the ground-truth
dataclass. Consumers should treat unknown keys as opaque.
phase : Phase, optional
Capture phase the frame belongs to. Required for ESD-weighted
phase scoring.
difficulty : Difficulty, optional
ESD difficulty label of the (scene, phase) this sample
belongs to.
camera_pose : np.ndarray, optional
4 × 4 float64 SE(3) matrix (camera → world) sourced from the
T265 tracker. Used for the temporal-stability metric.
Sample( id: str, rgb: numpy.ndarray, ground_truth: Any, metadata: Optional[Dict[str, Any]] = None, phase: Phase | None = None, difficulty: Difficulty | None = None, camera_pose: numpy.ndarray | None = None)
@dataclass
class
VideoSample:
273@dataclass 274class VideoSample: 275 """One input unit for video tasks (Video Depth today, possibly more later). 276 277 Mirrors :class:`Sample` but carries sequence-shaped fields so a 278 video model can ingest the entire phase clip in one call. The 279 iteration unit is a ``(scene, phase)`` clip, not a frame, so the 280 cell-log key for video tasks is ``(model, task, scene, phase)`` 281 rather than ``(..., frame)``. 282 283 Shapes 284 ------ 285 id : ``"{scene}_{phase}"`` 286 rgb_seq : (T, H, W, 3) uint8 287 ground_truth : task-specific (e.g. :class:`VideoDepthGroundTruth`) 288 metadata : dict — must include ``frame_indices`` (T,) int32, 289 ``intrinsics`` (3, 3) float32, and optionally 290 ``ego_motion_seq`` (T, 4, 4) float64 from T265 291 phase : :class:`Phase` enum value for this clip 292 difficulty : :class:`Difficulty` of the ``(scene, phase)`` 293 camera_pose_seq : (T, 4, 4) float64, camera → world per frame, or 294 ``None`` if the task does not need pose context 295 296 The split from :class:`Sample` is deliberate: trying to bolt 297 sequence fields into a single Sample type would have made 298 every per-frame consumer guard against ``rgb_seq is None``. Two 299 explicit shapes keep adapters honest about which task they implement. 300 """ 301 302 id: str 303 rgb_seq: np.ndarray 304 ground_truth: Any 305 metadata: Dict[str, Any] | None = None 306 phase: Phase | None = None 307 difficulty: Difficulty | None = None 308 camera_pose_seq: np.ndarray | None = None
One input unit for video tasks (Video Depth today, possibly more later).
Mirrors Sample but carries sequence-shaped fields so a
video model can ingest the entire phase clip in one call. The
iteration unit is a (scene, phase) clip, not a frame, so the
cell-log key for video tasks is (model, task, scene, phase)
rather than (..., frame).
Shapes
id : "{scene}_{phase}"
rgb_seq : (T, H, W, 3) uint8
ground_truth : task-specific (e.g. VideoDepthGroundTruth)
metadata : dict — must include frame_indices (T,) int32,
intrinsics (3, 3) float32, and optionally
ego_motion_seq (T, 4, 4) float64 from T265
phase : Phase enum value for this clip
difficulty : Difficulty of the (scene, phase)
camera_pose_seq : (T, 4, 4) float64, camera → world per frame, or
None if the task does not need pose context
The split from Sample is deliberate: trying to bolt
sequence fields into a single Sample type would have made
every per-frame consumer guard against rgb_seq is None. Two
explicit shapes keep adapters honest about which task they implement.
VideoSample( id: str, rgb_seq: numpy.ndarray, ground_truth: Any, metadata: Optional[Dict[str, Any]] = None, phase: Phase | None = None, difficulty: Difficulty | None = None, camera_pose_seq: numpy.ndarray | None = None)
@dataclass
class
DepthPrediction:
@dataclass
class
VideoDepthPrediction:
316@dataclass 317class VideoDepthPrediction: 318 """A video depth model's output for one phase clip. 319 320 Shape 321 ----- 322 depth_map_seq : (T, H, W) float32, metres. Models that emit 323 affine-invariant (relative) depth should still return a 324 metric-scale tensor; the loader/runner will handle per-clip 325 scale-and-shift alignment before metric computation (canonical 326 Ranftl et al. 2020 procedure). Aligning per-frame would defeat 327 the temporal consistency metrics; aligning per-clip is the 328 standard convention for video-depth evaluation. 329 """ 330 331 depth_map_seq: np.ndarray
A video depth model's output for one phase clip.
Shape
depth_map_seq : (T, H, W) float32, metres. Models that emit affine-invariant (relative) depth should still return a metric-scale tensor; the loader/runner will handle per-clip scale-and-shift alignment before metric computation (canonical Ranftl et al. 2020 procedure). Aligning per-frame would defeat the temporal consistency metrics; aligning per-clip is the standard convention for video-depth evaluation.
@dataclass
class
DetectionPrediction:
@dataclass
class
SegmentationPrediction:
@dataclass
class
TrackletPrediction:
TrackletPrediction(tracks: Sequence[Tracklet])
tracks: Sequence[Tracklet]
@dataclass
class
VisualGroundingPrediction:
351@dataclass 352class VisualGroundingPrediction: 353 boxes: np.ndarray # N x 4 float 354 scores: np.ndarray # N float 355 labels: Sequence[str] | None = None
@dataclass
class
RelativePosePrediction:
@dataclass
class
SparseDepthPrediction:
@dataclass
class
KeypointCorrespondencePrediction:
372@dataclass 373class KeypointCorrespondencePrediction: 374 points0: np.ndarray # N x 2 375 points1: np.ndarray # N x 2 376 scores: np.ndarray | None = None
class
BenchmarkModel(abc.ABC):
379class BenchmarkModel(ABC): 380 """Abstract base class every RPX-compatible model must implement. 381 382 In practice, most users should **not** subclass this directly — 383 instead compose a :class:`rpx_benchmark.adapters.BenchmarkableModel` 384 from an input adapter, a model callable, and an output adapter. 385 ``BenchmarkableModel`` already implements :meth:`predict` and 386 :meth:`setup` correctly for you. 387 388 Subclass only when you need complete control over how samples 389 are routed to your model (e.g. true minibatching across GPU 390 devices). 391 392 Attributes 393 ---------- 394 task : TaskType 395 The task this model solves. Must be set by subclasses (either 396 at class level or in ``__init__``). The runner checks that 397 ``model.task == dataset.task`` before running. 398 depth_output_kind : Literal["metric", "relative"] 399 Only meaningful for depth tasks (``MONOCULAR_DEPTH`` and 400 ``VIDEO_DEPTH``). Declares whether the model emits depth in 401 physical units (``"metric"``, default — passes through the 402 runner unchanged) or up to an unknown scale and shift 403 (``"relative"`` — runner applies per-scene-phase scale+shift 404 alignment via :func:`~rpx_benchmark.metrics.depth_alignment.align_pred_to_gt_pooled` 405 before computing metrics, matching the Ranftl et al. 2020 406 protocol the paper §3.3 prescribes). 407 408 Adapters that wrap relative-depth models (Lotus-2, FE2E and 409 the DA-V2 relative variant) must set 410 this to ``"relative"``. Adapters wrapping metric models 411 (DA-Metric, Depth Pro, UniDepth V2, Metric3D V2, MoGe-2, 412 canonical HyDen metric, DepthLM, etc.) leave 413 the default. Ignored entirely for non-depth tasks. 414 415 Examples 416 -------- 417 Minimal subclass:: 418 419 class MyDepth(BenchmarkModel): 420 task = TaskType.MONOCULAR_DEPTH 421 # depth_output_kind defaults to "metric" 422 423 def setup(self): 424 self.net = load_my_checkpoint() 425 426 def predict(self, batch): 427 return [ 428 DepthPrediction(depth_map=self.net(s.rgb)) 429 for s in batch 430 ] 431 432 Relative-depth wrapper (Lotus-2, FE2E, MoGe-2, ...):: 433 434 class MyRelativeDepth(BenchmarkModel): 435 task = TaskType.MONOCULAR_DEPTH 436 depth_output_kind = "relative" 437 438 def setup(self): ... 439 def predict(self, batch): ... # returns up-to-scale depth 440 441 Composed via :class:`BenchmarkableModel`:: 442 443 bm = rpx.BenchmarkableModel( 444 task=TaskType.MONOCULAR_DEPTH, 445 input_adapter=MyInputAdapter(), 446 model=my_nn_module, 447 output_adapter=MyOutputAdapter(), 448 name="my_model", 449 ) 450 """ 451 452 task: TaskType 453 #: Depth output convention — ``"metric"`` (default) means the 454 #: prediction is in metres and goes straight to the metric 455 #: calculators. ``"relative"`` means the prediction is up to an 456 #: unknown scale + shift; the runner applies per-scene-phase 457 #: pooled scale-and-shift alignment via 458 #: :func:`~rpx_benchmark.metrics.depth_alignment.align_pred_to_gt_pooled` 459 #: before computing metrics. Ignored for non-depth tasks. 460 depth_output_kind: Literal["metric", "relative"] = "metric" 461 462 @abstractmethod 463 def setup(self) -> None: 464 """Load checkpoints, warm CUDA, and do any other one-time init. 465 466 The runner calls this exactly once before iterating the 467 dataset, unless ``BenchmarkRunner(call_setup=False)`` was 468 passed — in which case the caller is responsible. 469 """ 470 471 @abstractmethod 472 def predict(self, batch: Sequence[Sample]) -> Sequence[Any]: 473 """Run inference on a batch of samples. 474 475 Parameters 476 ---------- 477 batch : sequence of Sample 478 One or more samples. Length equals ``dataset.batch_size`` 479 except possibly for the final tail batch. 480 481 Returns 482 ------- 483 sequence 484 One task-specific Prediction dataclass per input sample, 485 in the same order. The prediction dataclass must match 486 what :class:`MetricSuite` expects for this task. 487 488 Raises 489 ------ 490 ModelError 491 (By convention) when a sample cannot be processed. The 492 runner surfaces it as a clean error rather than a stack 493 trace. 494 """
Abstract base class every RPX-compatible model must implement.
In practice, most users should not subclass this directly —
instead compose a rpx_benchmark.adapters.BenchmarkableModel
from an input adapter, a model callable, and an output adapter.
BenchmarkableModel already implements predict() and
setup() correctly for you.
Subclass only when you need complete control over how samples are routed to your model (e.g. true minibatching across GPU devices).
Attributes
task : TaskType
The task this model solves. Must be set by subclasses (either
at class level or in __init__). The runner checks that
model.task == dataset.task before running.
depth_output_kind : Literal["metric", "relative"]
Only meaningful for depth tasks (MONOCULAR_DEPTH and
VIDEO_DEPTH). Declares whether the model emits depth in
physical units ("metric", default — passes through the
runner unchanged) or up to an unknown scale and shift
("relative" — runner applies per-scene-phase scale+shift
alignment via ~rpx_benchmark.metrics.depth_alignment.align_pred_to_gt_pooled()
before computing metrics, matching the Ranftl et al. 2020
protocol the paper §3.3 prescribes).
Adapters that wrap relative-depth models (Lotus-2, FE2E and
the DA-V2 relative variant) must set
this to ``"relative"``. Adapters wrapping metric models
(DA-Metric, Depth Pro, UniDepth V2, Metric3D V2, MoGe-2,
canonical HyDen metric, DepthLM, etc.) leave
the default. Ignored entirely for non-depth tasks.
Examples
Minimal subclass::
class MyDepth(BenchmarkModel):
task = TaskType.MONOCULAR_DEPTH
# depth_output_kind defaults to "metric"
def setup(self):
self.net = load_my_checkpoint()
def predict(self, batch):
return [
DepthPrediction(depth_map=self.net(s.rgb))
for s in batch
]
Relative-depth wrapper (Lotus-2, FE2E, MoGe-2, ...)::
class MyRelativeDepth(BenchmarkModel):
task = TaskType.MONOCULAR_DEPTH
depth_output_kind = "relative"
def setup(self): ...
def predict(self, batch): ... # returns up-to-scale depth
Composed via BenchmarkableModel::
bm = rpx.BenchmarkableModel(
task=TaskType.MONOCULAR_DEPTH,
input_adapter=MyInputAdapter(),
model=my_nn_module,
output_adapter=MyOutputAdapter(),
name="my_model",
)
task: TaskType
@abstractmethod
def
setup(self) -> None:
462 @abstractmethod 463 def setup(self) -> None: 464 """Load checkpoints, warm CUDA, and do any other one-time init. 465 466 The runner calls this exactly once before iterating the 467 dataset, unless ``BenchmarkRunner(call_setup=False)`` was 468 passed — in which case the caller is responsible. 469 """
Load checkpoints, warm CUDA, and do any other one-time init.
The runner calls this exactly once before iterating the
dataset, unless BenchmarkRunner(call_setup=False) was
passed — in which case the caller is responsible.
471 @abstractmethod 472 def predict(self, batch: Sequence[Sample]) -> Sequence[Any]: 473 """Run inference on a batch of samples. 474 475 Parameters 476 ---------- 477 batch : sequence of Sample 478 One or more samples. Length equals ``dataset.batch_size`` 479 except possibly for the final tail batch. 480 481 Returns 482 ------- 483 sequence 484 One task-specific Prediction dataclass per input sample, 485 in the same order. The prediction dataclass must match 486 what :class:`MetricSuite` expects for this task. 487 488 Raises 489 ------ 490 ModelError 491 (By convention) when a sample cannot be processed. The 492 runner surfaces it as a clean error rather than a stack 493 trace. 494 """
Run inference on a batch of samples.
Parameters
batch : sequence of Sample
One or more samples. Length equals dataset.batch_size
except possibly for the final tail batch.
Returns
sequence
One task-specific Prediction dataclass per input sample,
in the same order. The prediction dataclass must match
what MetricSuite expects for this task.
Raises
ModelError (By convention) when a sample cannot be processed. The runner surfaces it as a clean error rather than a stack trace.
508def validate_prediction(task: TaskType, prediction: Any, sample: Sample | None = None) -> None: 509 """Validate a Prediction dataclass's shape and type for a given task. 510 511 Parameters 512 ---------- 513 task : TaskType 514 Task the runner is evaluating. 515 prediction : Any 516 Prediction dataclass the model just returned. 517 sample : Sample, optional 518 The sample the prediction was produced for; used for shape 519 cross-checks (e.g. segmentation mask vs RGB size). 520 521 Raises 522 ------ 523 ModelError 524 If the prediction is the wrong type or the wrong shape for the 525 task. 526 """ 527 from .exceptions import ModelError 528 529 def _type_error(expected: str) -> "ModelError": 530 return ModelError( 531 f"{task.value} models must return {expected}, got {type(prediction).__name__}", 532 hint=f"Check your model's predict() return type for {task.value}.", 533 ) 534 535 if task == TaskType.MONOCULAR_DEPTH: 536 if not isinstance(prediction, DepthPrediction): 537 raise _type_error("DepthPrediction") 538 _check_shape(prediction.depth_map, 2, "Depth map") 539 return 540 541 if task in (TaskType.OBJECT_DETECTION, TaskType.OPEN_VOCAB_DETECTION): 542 if not isinstance(prediction, DetectionPrediction): 543 raise _type_error("DetectionPrediction") 544 boxes = prediction.boxes 545 _check_shape(boxes, 2, "Detection boxes") 546 if boxes.shape[1] != 4: 547 raise ModelError( 548 f"Detection boxes must have shape [N, 4]; got {boxes.shape}", 549 ) 550 if len(prediction.scores) != len(boxes) or len(prediction.labels) != len(boxes): 551 raise ModelError( 552 "DetectionPrediction boxes, scores, and labels must all have the same length.", 553 details={ 554 "n_boxes": len(boxes), 555 "n_scores": len(prediction.scores), 556 "n_labels": len(prediction.labels), 557 }, 558 ) 559 return 560 561 if task == TaskType.OBJECT_SEGMENTATION: 562 if not isinstance(prediction, SegmentationPrediction): 563 raise _type_error("SegmentationPrediction") 564 _check_shape(prediction.mask, 2, "Segmentation mask") 565 if sample and prediction.mask.shape != sample.rgb.shape[:2]: 566 raise ModelError( 567 f"Segmentation mask shape {prediction.mask.shape} must " 568 f"match the RGB spatial size {sample.rgb.shape[:2]}", 569 hint="The OutputAdapter should resize masks to the sample's " 570 "original H x W before returning.", 571 ) 572 return 573 574 if task == TaskType.OBJECT_TRACKING: 575 if not isinstance(prediction, TrackletPrediction): 576 raise _type_error("TrackletPrediction") 577 return 578 579 if task == TaskType.VISUAL_GROUNDING: 580 if not isinstance(prediction, VisualGroundingPrediction): 581 raise _type_error("VisualGroundingPrediction") 582 return 583 584 if task == TaskType.RELATIVE_CAMERA_POSE: 585 if not isinstance(prediction, RelativePosePrediction): 586 raise _type_error("RelativePosePrediction") 587 return 588 589 if task == TaskType.SPARSE_DEPTH: 590 if not isinstance(prediction, SparseDepthPrediction): 591 raise _type_error("SparseDepthPrediction") 592 return 593 594 595 if task == TaskType.KEYPOINT_MATCHING: 596 if not isinstance(prediction, KeypointCorrespondencePrediction): 597 raise _type_error("KeypointCorrespondencePrediction") 598 return 599 600 raise ModelError(f"Unsupported task: {task}")
Validate a Prediction dataclass's shape and type for a given task.
Parameters
task : TaskType Task the runner is evaluating. prediction : Any Prediction dataclass the model just returned. sample : Sample, optional The sample the prediction was produced for; used for shape cross-checks (e.g. segmentation mask vs RGB size).
Raises
ModelError If the prediction is the wrong type or the wrong shape for the task.