Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction

Yangxiao Lu1          Ninad Khargonkar1          Zesheng Xu1          Charles Averill1          Kamalesh Palanisamy1          Kaiyu Hang2          Yunhui Guo1          Nicholas Ruozzi1          Yu Xiang1
1The University of Texas at Dallas         2Rice University

Robotics: Science and Systems (RSS), 2023

Abstract

We introduce a novel robotic system for improving unseen object instance segmentation in the real world by leveraging long-term robot interaction with objects. Previous approaches either grasp or push an object and then obtain the segmentation mask of the grasped or pushed object after one action. Instead, our system defers the decision on segmenting objects after a sequence of robot pushing actions. By applying multi-object tracking and video object segmentation on the images collected via robot pushing, our system can generate segmentation masks of all the objects in these images in a self-supervised way. These include images where objects are very close to each other, and segmentation errors usually occur on these images for existing object segmentation networks. We demonstrate the usefulness of our system by fine-tuning segmentation networks trained on synthetic data with real-world data collected by our system. We show that, after fine-tuning, the segmentation accuracy of the networks is significantly improved both in the same domain and across different domains. In addition, we verify that the fine-tuned networks improve top-down robotic grasping of unseen objects in the real world.

Paper


BibTex

Please consider citing the paper if it helps in your research.


@article{lu2023self,
  title={Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction},
  author={Lu, Yangxiao and Khargonkar, Ninad and Xu, Zesheng and Averill, Charles and Palanisamy, Kamalesh and Hang, Kaiyu and Guo, Yunhui and Ruozzi, Nicholas and Xiang, Yu},
  journal={arXiv preprint arXiv:2302.03793},
  year={2023}
}
          

Dataset

Feel free to check out the dataset using the Download Link (UT Dallas Box, no login required). The dataset download is a zip file containing the train and test folders. Both of them have data for multiple pushing trials as sub-folders, with the data for rgb image, depth, initial segmentation, refined segmentation, meta-data etc.

Creative Commons License
This dataset is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Code

GitHub Link for segmentation model training, evaluation and real-world ROS testing.
GitHub Link for object segmentation dataset generation via multi-object tracking and mask propapagtion based on video object segmentation.

Contact

Send any comments or questions to Yangxiao Lu: Yangxiao's UT Dallas email id


Last updated on 07-February-2023 | Template adapted from: dex-ycb.github.io