Solution Presentation: Automatica 2023 - Interactive Bin Picking with Custom GUI

In this post we will look at the bin picking application with a focus on the custom GUI that was displayed at the YASKAWA booth at Automatica23 in Munich, Germany.

Fig. 1: Bin picking cell as seen at the trade fair

Task Description

The scene consists of the robot, the bin for picking (filled with table tennis balls), and a platform to place the balls, as can be seen in Fig. 1 and Fig. 2. How the balls should be placed is specified in a Guided User Interface (GUI) that was custom-built for this application. The user chooses a pattern and presses Start. The robot then picks a ball of the desired color and places it at the next position in the pattern. If the pattern is complete or the user presses Stop and Start again, the robot takes all balls from the platform and drops them in the bin for picking, effectively resetting the scene.

Fig. 2: Simulated robot picking balls and placing them in specified pattern


  • Software Version: 1.7.1
  • Camera: Mech-Eye Nano V3 (SDK Version 2.1.0)
  • Robot: YASKAWA HC10DTP



The Mech-Vision project follows the typical structure of a DL supported project:

  1. Camera Triggering
  2. Point Cloud Segmentation and Object Class Recognition with DL (Instance Segmentation)
  3. Model Matching
  4. Pose Transformation

Fig. 3: Mech-Vision overall workflow

Tips and Tricks

Model Matching

In this application, DL is used to detect the table tennis instances and their label (blue or white). The step Deep Learning Model Package Inference returns the 2D mask for each instance and its label. The model was custom trained on only ten images. In a subsequent step, Apply Masks to Point Cloud, the 2D mask is used to segment the Point Cloud into the parts of each ball. Note that instead of DL, Clustering and Color Thresholding could have also been used to achieve the segmentation.
Afterwards, the typical matching procedures, 3D Coarse Matching V2, and 3D Fine Matching Lite, are used to match the model of the ball (see Fig. 4). Since each segmented point cloud only contains a single ball, the parameter Maximum Number of Detected Poses should be set to one.
Attention should be paid to setting the Confidence Threshold parameter. This value has to be set to 0 in the 3D Fine Matching Lite step, and subsequently, the Compare Value step has to be used to filter out low confidence results. The reason is that the label of the ball is already determined before the matching step. For each of the N point clouds, a matching result is returned if the confidence threshold is set to 0. Therefore, the pose list of the fine matching step has a length of N. If you were to filter out the low confidence values during the matching step, you would lose the one-to-one correspondence of the pose list and the label list. This is the reason why you need additional steps to apply the confidence threshold.

Fig. 4: Mech-Vision matching workflow

Pose Filtering

In order to pick only a ball of the correct color for the next position in the placing pattern, the step Validate Labels and Output Flags is used to filter the poses by the label, as shown in Fig. 2. To achieve this, the reference labels are set externally using an adapter. How this can be done in the adapter will be explained later in this post.

Pose Transformation

Independent of the matching result, the z-axis of the ball should point straight up. To achieve this, there are multiple possibilities, two of which are shown in Fig. 5. The orientation of the pose is set via a desired quaternion, resulting in an alignment of the object’s z-axis with the z-axis of the camera or robot frame, depending on whether the pose was transformed to the robot frame beforehand or not.
Secondly, the pose (defined at the ball’s center) has to be translated to the ball’s surface. Alternatively, the TCP (Tool Center Point) of the gripper could also be adjusted such that the TCP is not at its tip but at an offset (the ball’s radius), making the translation redundant.

Fig. 5: Two possibilities to transform the poses


In this application, the logic of Mech-Viz is partly controlled by an adapter. The adapter is a Python script that sends and receives control and status commands from Mech-Viz and can thereby be used for more complex projects. The script used for this case is shown in the next section called Adapter.
A branch_by_msg step is used to differentiate between normal operation (branch 0) and the operation that should take place once the user hits the stop button (branch 1). During normal operation, first, all balls on the platform are removed, and afterward, the balls are picked from the pick box one by one and placed on the platform.
As you may have noticed in Fig. 6, Notify steps are used. The Notify step is used to send a message to the adapter. In this case, different functions of the adapter are called when the adapter receives the message.
Once the pattern is complete (Check Completion evaluates to True), the loop is exited, and the Viz program is terminated. It will be restarted once the user defines a new pattern and presses Start.

Fig. 6: Mech-Viz overall workflow and the workflow for picking and placing

Adapter and GUI


To use an adapter (and a UI) it is necessary to define the path of the parent directory of the adapter in Mech-Center/Deployment Settings/Mech-Interface. Once you press Start Interface in Mech-Center the adapter (and the UI) are started. it is also possible to use an adapter without a UI.
The adapter directory should look like this:


Important is that the __init__ file has this structure:

import sys
import os
from Adapter.adapter import AIAdapter
from Adapter.widget import CustomGui

_adater_path = os.path.abspath(os.path.join(os.path.dirname(__file__)))
if _adater_path not in sys.path:

adapter_dict = {os.path.basename(_adater_path): AIAdapter}
adapter_widget_dict={os.path.basename(_adater_path): CustomGui}


The GUI is based on a Pyside2 Widget and is created like this:

... # custom imports here
from interface.adapter import AdapterWidget

class CustomGui(AdapterWidget):
    def __init__(self):
        ... # custom code here
    def pattern_completed(self):
        ... # custom code here

    def after_set_adapter(self):
        ... # custom code here

Therefore, you can develop a UI with all the possibilities that Pyside2 offers. Note that the CustomGUI class has to inherit from AdapterWidget. When the user interface in Mech-Center is started, the adapter is assigned to the UI instance. Afterwards, the method after_set_adapter is called, where you should connect signals of the adapter with the signals of the Widget. For example, in this case, whenever the signal pattern_completed_signal of the adapter is triggered (which happens once all balls are placed) the method pattern_completed defined in the widget is called. The UI developed for this application is shown in Fig. 7.

Fig. 7: Custom GUI of the application with and without dark mode


Importing the required modules:

import logging
import json
from functools import partial

from PySide2.QtCore import Signal, QTimer

from interface.adapter import TcpServerAdapter
from import NotifyService, register_service

To receive messages from the Notify step in Mech-Viz it is necessary to define a service class:

class NotifyLogic(NotifyService):
    service_name = "Viz_Notify"

    def __init__(self, adapter):
        self.adapter = adapter

    def handle_message(self, msg):
        if msg.lower() == "next":

        if msg.lower() == "index_increase":
            self.adapter.index += 1
            self.adapter.msg_signal.emit(logging.DEBUG, f'index_increase:{self.adapter.index}')

        if msg.lower() == "finished":
            self.adapter.index = 0

The method handle_message of this service class is called every time a Notify step in Mech-Viz is exectued. Note that in the Notify block, you have to specify the service name of the service class that receives the messages, Viz_Notify in this case.
Depending on the message (a string) a different routine will be executed.
adapter.msg_signal.emit can be used to log output to the Mech-Center console.

Finally, the control logic is defined in the AIAdapter class that inherits from the TcpServerAdapter:

class AIAdapter(TcpServerAdapter):
    robot_name = None
    pattern_completed_signal = Signal(bool)

    def __init__(self, address):
  'Services succesfully registered!')
        except Exception as e:
            logging.error(f"Services could not be succesfully registered! \n {e}")
        self.set_pattern([])  # empty init

    def set_current_pick_color(self):
        visionStepName = "SetBallColor"
        visionProjectName = "YASKAWA"
        color = str(self.pattern[self.index]) # get color

        msg = {"name": visionStepName, "values": {"strings": color}}
        self.msg_signal.emit(logging.DEBUG, "Blue" if color == '0' else "White")
        self.set_step_property(msg, visionProjectName)

    def set_branch(self, name, out_port):
            info = {"out_port": out_port, "other_info": []}
            msg = {"name": name,
                   "values": {"info": json.dumps(info)}}
            QTimer.singleShot(500, partial(self.set_task_property, msg)) # delay
            self.msg_signal.emit(logging.INFO, f"Set port {out_port} of {name}")
        except Exception as e:

    def set_pattern(self, pattern):
        self.pattern = pattern
        self.index = 0 # reset

    def _register_service(self):
        self.adapter_service = NotifyLogic(self)
        self.server, _ = register_service(self.hub_caller, self.adapter_service)

    def _unregister_service(self):
        if self.adapter_service:

    def close(self):

Importantly, the pattern_completed_signal has to be defined as a class variable (a PySide2 peculiarity), and the notify service has to be registered in the _register_service method.
The output port of the branch_by_msg step can be set as shown above. Note that QTimer is used to execute the function with a delay. The reason is that the branch is set immediately after the ‘start Mech-Viz’ command is sent. However, Mech-Viz does not immediately start after the command but only starts running with some small delay. Introducing a small delay ensures that the set branch command is sent after Mech-Viz is running, resolving this issue.
In order to pick only balls of a certain color, the method set_step_property is used. This method overwrites values that are set in the specified step (‘SetBallColor’) of the specified Vision project (‘YASKAWA’ here). Therefore, before the camera is triggered, the color for the next position in the pattern is set.

  • This is helpful.
  • This is not helpful.
0 voters