For using the deep learning model to recognize before the subsequent matching, I have attempted the following operations, but failed to pass the label. How should I resolve this?
How to pass the label recognized by the DL model to 3D?
How to pass the label recognized by the DL model to 3D?
- The fine match step can encompass the original label.
- The deep learning label is generally used to differentiate between different workpieces or the front and back. Whatever the case, it requires an additional match step to replace and use someone else.
Not sure if it’s exactly applicable to your scenario but I have encountered one scenario where I had problems with directly mapping labels from deep learning to model matching. Thought it’s worth a share.
Scenario:
Picked object:
Single rectangular object shape but comes in different colours. (Red, blue, yellow)
Plan of approach:
- Deep learning to recognise the objects and label it by its colour (e.g. 0 for red, 1 for blue…)
- Followed by matching to improve picking accuracy especially for tilted and overlapped parts.
Problem encountered: ‘3D Coarse matching V2’ may missed out /eliminate certain masked images from the point cloud and it does not have an input for labels. This results in mismatched with labels and masked images.
E.g. Deep learning outputs 3 masked images; each of different label corresponding to its colour (red, blue ,yellow). However, let’s say coarse matching did not manage to recognise the first masked image (red colour) and only outputs the latter 2(blue, yellow). Hence, you will have three labels but only two poses when inputting to the 3D fine matching step, resulting in vector size mismatch error.
Note: This would not be a problem if your model matching is different for each label.
Workaround solution:
- Filter the masked images from deep learning by label. (e.g. Filter red colour masked images only and remove other colours) (Image 1)
- Perform model matching (ignore labels as you’re dealing with one colour only). (Image 2)
- Add back labels to the final poses. (Image 3)
- Do this for all colours
Note: Make sure that ‘Continue Given No Output’ is ticked for 3D fine matching and remove overlapped objects steps especially if you’re merging your data together later. (image 4) Otherwise, mech-vision will stop processing further steps if certain labels/colours are missing.
Image 1 Doing similar model matching for each label
Image 2 Filter by colour/label
Image 3 Add back the label after model matching
Image 4 ‘Continue given no output’ under execution flags
3 Likes
Hello Rene, thank you so much for sharing the solution in such great detail! It is a very clever strategy to map the labels from DL to 3D matching. I have tried your approach, and it works just fine now.