Under what circumstances do we use deep learning in our solutions?

Could you give some examples?

When the point cloud of the workpiece is of poor quality in a project, it is generally necessary to use a combination of deep learning and 3D point cloud processing for workpiece recognition, localization, and grasping. Here are a few examples:

1.When key features of the workpiece are missing from the point cloud:

In cases where crucial features of the workpiece are absent from the point cloud data

2D image information:

The red boxes in the image represent the front view of workpieces A and B, while the blue box shows the reverse side of the A and B frames. The arrows in the image point to the main feature regions that distinguish the two types of workpieces.

3D point cloud information:

As can be observed in the screenshot of the point cloud below, the features that differentiate between the types of workpieces cannot be obtained from the point cloud.

Deep learning optimization:

The project benefits from high-quality 2D images, making it possible to employ deep learning directly for segmentation and differentiation of workpiece types.
The type of deep learning utilized in this project is instance segmentation. By assigning appropriate classification labels to various types of workpieces, the instance segmentation process extracts individual workpiece masks and simultaneously provides the corresponding category label for each identified workpiece.

2.Severe point cloud incompleteness:

The workpiece’s point cloud data exhibits significant gaps and missing portions. During 3D point cloud matching, mismatches can occur, leading to substantial deviations in grasping pose.

2D image information:

The reflective workpieces in the image are closely packed and numerous. However, the edges and shape features of the workpieces’ can be clearly discerned.

3D point cloud information:

Due to the reflective nature of the workpieces, point cloud data may exhibit gaps, with the primary gaps occurring along the axial direction of the workpieces.

Deep learning optimization:

In this project, an instance segmentation model is employed to recognize the workpieces, generating corresponding masks. Subsequently, the point cloud corresponding to each mask is extracted.

3.Virtually no point cloud for the workpieces:

In projects of this type, conventional methods for recognition and grasping are ineffective. Instead, it’s necessary to employ deep learning to generate masks and subsequently compute poses for point cloud generation. Note: High-quality 2D images are essential.

2D image information:

The waveform sleeves often consist of reflective components and are tightly packed within the material container.

3D point cloud information:

Upon examining the point cloud image, it is evident that the point cloud for this particular workpiece is highly unstable. There is a significant possibility of encountering instances where the workpiece has absolutely no points in the point cloud.

Deep learning optimization:

Despite the workpieces being reflective in nature, the edge information of the workpieces is clear in the 2D images. Therefore, utilizing deep learning for workpiece recognition can achieve stable and reliable results in this project.