We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on the task of pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible for researchers and practitioners. We also provide our experimental results and analyses for model-based and model-free 6D robotic grasping on the benchmark, where representative algorithms for object perception, grasping planning and motion planning are evaluated. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods.
20 scenes in our SceneReplica benchmark with 5 YCB objects in each scene
The process of replicating a scene in the real world. The reference scene image is overlaid to the real camera image to guide how to place objects into the real-world scene.
# | Perception | Grasp Planning | Motion Planning | Control | Ordering | Grasping Type | Pick & Place Success | Grasping Success | Videos |
---|---|---|---|---|---|---|---|---|---|
3 | GDRNPP [9] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Based | 66/100 | 69/100 | |
3 | GDRNPP [9] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Based | 62/100 | 64/100 | |
7 | MSMFormer [8] | Contact-graspnet [7] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Free | 61/100 | 70/100 | |
5 | UCN [5] | Contact-graspnet [7] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Free | 60/100 | 63/100 | |
5 | UCN [5] | Contact-graspnet [7] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Free | 60/100 | 64/100 | |
1 | PoseRBPF [1] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Based | 59/100 | 59/100 | |
1 | PoseRBPF [1] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Based | 58/100 | 64/100 | |
7 | MSMFormer [8] | Contact-graspnet [7] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Free | 57/100 | 65/100 | |
8 | MSMFormer [8] | Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Free | 56/100 | 59/100 | |
2 | PoseCNN [4] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Based | 48/100 | 50/100 | |
4 | UCN [5] | GraspNet [6] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Free | 43/100 | 46/100 | |
9 | DexNet 2.0 [10] | DexNet 2.0 [10] | OMPL [3] | MoveIt | Algorithmic | Model-Free | 43/100 | 51/100 | |
2 | PoseCNN [4] | GraspIt! [2] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Based | 38/100 | 44/100 | |
6 | MSMFormer [8] | GraspNet [6] + Top-Down | OMPL [3] | MoveIt | Near-to-Far | Model-Free | 38/100 | 41/100 | |
4 | UCN [5] | GraspNet [6] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Free | 37/100 | 40/100 | |
6 | MSMFormer [8] | GraspNet [6] + Top-Down | OMPL [3] | MoveIt | Fixed Random | Model-Free | 36/100 | 41/100 |
@article{khargonkar2023scenereplica,
title={SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes},
author={Ninad Khargonkar and Sai Haneesh Allu and Yangxiao Lu and Jishnu Jaykumar P and Balakrishnan Prabhakaran and Yu Xiang},
journal={arXiv preprint arXiv:2306.15620},
year={2023}}
Send any comments or questions to Ninad | Sai:
ninadarun.khargonkar@utdallas.edu | saihaneesh.allu@utdallas.edu
This work was supported in part by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005.