T6 WALKTHROUGH
In-context VQA¶
Takes a reference crop of the object (from its single-object capture), then the target image, plus a question, and returns a box in the target image.
The same scene 049 target, with a reference crop of the lion from its single-object capture. In a real run the crop comes from the manifest, with a pinned revision and hash.
Connect your model¶
Put your prediction function in my_model.py. Load the model or API client once, at module level, rather than on every call.
def predict_vqa(image_paths, prompt, max_new_tokens, output_kind):
assert len(image_paths) == 2 # reference crop, then target
return client.generate(images=image_paths, prompt=prompt,
max_new_tokens=max_new_tokens)
image_paths is (reference_crop, target); send them to the model in that order. Box coordinates refer to the target image. A model that accepts only one image can't run T6.
Run the benchmark¶
Install the toolkit with pip install 'rpx-benchmark[hub,schemas]'. The smoke test checks the install; the second tab scores your model.
This runs the real evaluator on a few synthetic inputs with a placeholder model. Nothing is downloaded and no weights are needed. The output directory must be new or empty.
Run this from the directory that contains my_model.py. Without --manifest, it scores the 16 T6 questions packaged with the toolkit, pinned to the dataset release, and downloads only their images.
python -m rpx_benchmark.examples.benchmark_tasks \
--task T6 \
--model my_model:predict_vqa \
--model-name my-model \
--model-revision exact-checkpoint-or-api-version \
--output results/my-model/T6
The packaged questions are a quick-start set, not the paper's full split. To score more, build a canonical manifest (how) and pass it with --manifest and --image-cache. Every row of a manifest you pass is scored, and --max-samples is refused, so cut the manifest itself for a short run. Keep each row's question, box, image locator and reference metadata intact.
Read the results¶
run_config.jsonModel and evaluation settingspredictions.jsonlRaw responses for each questionresult.jsonScored metrics and failure countsCheck that the number of scored samples matches what you asked for. Keep the per-sample records: Φ and 𝒥min are computed from them. Latency and memory are measured separately, with the hardware profiler.
Scoring and protocol¶
T6 needs an in-context manifest. The evaluator rebuilds each reference crop from its stored crop box, checks its SHA-256, and loads both images at the revisions pinned in the manifest. Don't substitute the full single-object frame or a different object, and don't swap the two images.
The box is [x1, y1, x2, y2] normalized to 0–1000 in the target image. Parsing and metrics are the same as T5, with the in-context question types.
Before a full run
Check units, shapes, input order and sample IDs on a small run first. Pin the model revision, and keep the split and preprocessing the same for every model you compare. The smoke test only exercises the evaluator; your model still needs its own environment and weights.
Workflow figure
T6 workflow (SVG): the inputs, adapter and scoring stages for this task.
