- Solution Architecture
- Setting up the Edge Device
- Download the Required ADLINK Edge Profile with Profile Builder
- Deploying the Profile
- Viewing the Inference Results
- Acting on Inference Results
Solution Architecture
The ADLINK Edge Intel Arc Welding Solution is comprised of several applications which communicate via the ADLINK Data River. Together these applications ingest video from cameras, perform inferencing on the captured images and output the inference results for other applications to consume.
Optionally, additional ADLINK Edge applications can be added to the solution to enable bi-directional communication with OT devices such as PLCs.
The solution monitors a video stream and performs automatic visual detection of weld defects and can then act on their occurrence if required. An example action would be to stop the weld automatically to reduce the costs associated with wasted material.
The use case of the ADLINK Edge Intel Arc Welding Solution can be further broken down into the following:
- Stream video frames from a camera mounted to a robotic arc welder to monitor weld sessions.
- Detect weld defects by passing the video frames into an inference engine running a model. The model has been trained to detect three weld states: no_weld, normal_weld, and porosity_weld (i.e. a weld defect).
- Record weld sessions automatically as a video or as a series of images. Recording begins when a configurable trigger condition is met, such as when a weld begins, or when a weld defect is detected. Frame-by-frame inference data can also be stored in text files.
- Store weld session data as videos, images, and inference data. This can be either stored locally or transferred automatically to an AWS S3 bucket or an FTP server.
- Act on weld states, when a specific weld state occurs, an actionable signal can be sent to the robotic arc welder via Modbus or OPC-UA. For example, when a weld defect is detected, a signal can be sent to the arc welder to stop welding.
The core applications are:
- Deep Learning Streamer: Ingests video frames from a choice of multiple sources, then performs inferencing on those frames. Inference results are published back to the Data River for downstream applications to consume. Also provides the ability to save videos and images when a configurable trigger condition is met. Various triggers can be specified to control which frames are captured. For example, the application can be configured to capture frames classified as anomalous and/or having a confidence value below a certain threshold.
- Training Manager: Monitors a directory for images, videos, and inference result files captured by Deep Learning Streamer and transfers them to either AWS S3 or an FTP server.
- Node-RED: Provides a browser-based graphical flow editor which allows users to quickly wire together nodes to form an application. Nodes are provided to read and write samples from the Data River. An example dashboard application is included with the solution.
Additionally, the VMLink desktop application forms part of the solution and can be used to:
- View video streams from the Data River with inference results overlaid.
- Configure camera settings (where supported).
- Capture frames to a local folder with the ability to optionally synchronise the captured images to AWS S3.
Setting up the Edge Device
Edge Device Requirements
Intel OpenVino supports only x86 based machines so the target edge device must be of this architecture in order to deploy the profile. Detailed information on hardware requirements can be found on the Intel OpenVino website.
Ubuntu 18.04 LTS is recommended as the base operating system of your edge device.
Installing ADLINK Edge Profile Builder
ADLINK Edge Profile Builder can be installed on either the edge device or the PC used for development. To install ADLINK Edge Profile Builder follow the relevant guide below for your OS.
- Ubuntu 18.04 & NVIDIA Jetpack: Installing ADLINK Edge Profile Builder on Ubuntu 18.04 and JetPack
- Windows: Installing or Updating Edge Profile on a Windows PC
Installing VMLink
VMLink is a useful tool for viewing video streams along with associated inference results. It can also be used to collect images for training AI models. VMLink is currently available for Ubuntu 18.04 on x86 and ARM64.
For more information on installing VMLink please see How to Install VMLink on Ubuntu 18.04 LTS.
Installing IDS Software Suite on an Edge Device
The ADLINK Edge Intel Arc Welding Solution has been tested using IDS uEye cameras, as such, instructions in this guide will assume that you are using an IDS uEye camera. If you require assistance with camera selection, configuration, and mounting options, please contact ADLINK for further information.
In order to utilise an IDS uEye camera within the solution, you must download and install the IDS Software Suite for your specific camera model from the IDS website, this can be found by clicking here.
Download the Required ADLINK Edge Profile with Profile Builder
Before the ADLINK Edge applications making up the solution can be deployed, the applications must first be configured using ADLINK Edge Profile Builder.
To assist with this ADLINK provides the following template profile for download within Profile Builder:
- intel-arc-welding-x86
- Includes the ADLINK Edge Deep Learning Streamer Inference Engine application as well as other applications for reporting the results.
Downloading the Profiles
From within a project in ADLINK Edge Profile Builder click on Add profile and select Download a profile. Click the Next button.
Ensure the ADLINK-Templates repository is selected and select the intel-arc-welding-x86 profile. Click Next.
The profile will now be downloaded.
When the download is complete the profile will be available to edit in Profile Builder.
Deploying the Profile
Configuring the Profile
The profile includes the following applications:
- ADLINK Frame Streamer
- ADLINK Edge Deep Learning Streamer Inference Engine
- ADLINK Edge Training Manager
- Node-RED with ADLINK Data River nodes
Other ADLINK Edge and Docker applications can be added to the profile as required. For example, the ADLINK Edge Modbus Connect application can be added to support connections to Modbus devices. Please note these additional applications may require separate licenses to be purchased.
Frame Streamer Application
The Frame Streamer (aea-frame-streamer) application is included as part of the profile in order to stream a demo welding video to the DataRiver. It is pre-configured to stream the /videos/weld.mp4 video file to weldStream1. If you choose to utilise a uEye camera, remove this app from the profile.
To remove Frame Streamer from the profile:
- Click the three dots in the top right corner of the frame-streamer app tile
- Click “Delete Application” then “Delete forever”
Deep Learning Streamer Inference Engine Application
The Deep Learning Streamer (aea-deep-learning-streamer) application ingests video frames from a number of different sources and performs inferencing on those frames. It also provides the ability to record videos and images to the file system when configurable trigger events occur.
Configuring the input video stream
The profile comes pre-configured to ingest video frames from the DataRiver on weldStream1. If you wish to utilise a uEye camera the following configuration settings must be changed:
- InputsOption: IDSuEye
- camera: 0 (Change as applicable if your camera ID is different)
- configurationPath: /camera-configs/config.ini (optional)
The configurationPath setting above allows you to specify a uEye camera configuration ini file. This file can be exported from IDS Camera Manager. This setting is optional, but if you choose to set a configuration path, the corresponding file should be added to the application’s profile by adding the file to /camera-configs/config.ini using the Files tab.
Configuring event capture
The Captures section configures the conditions which trigger video or image recording. The default configuration included with the profile will capture video files and inference metadata whenever a normal_weld or porosity_weld is detected, recording the entirety of each weld session automatically. It will store these files in /adlinkedge/capture/weldStream1 within the container. This path is mounted to the local file system of the profile at ./capture/weldStream1.
If you wish to change the default configuration the following settings are of interest:
- Capture
- Format
- FormatOption: The default for the profile is video, but you can change this to images if you prefer to store one image file per video frame.
- EventCapture
- TimeBeforeEvent: The number of milliseconds before an event was detected to include in the output video or images.
- TimeAfterEvent: The number of milliseconds after an event was detected to include in the output video or images.
- EventFrameIndicator: Set this to false if you don’t want to have a box drawn around detections in the output video or images.
- Format
- Triggers: The trigger defines the conditions when a capture should start. You may want to configure additional parameters such as minimum confidence at which the capture will start.
- CategoryLabels: The profile by default will trigger when an inference result with either a porosity_weld or normal_weld label occurs on a video frame. You could, for example, remove the normal_weld label which would make captures only occur when a weld defect is detected.
Training Manager Application
The Training Manager (aea-training-manager) application monitors a directory for images, videos, and inference result files captured by Deep Learning Streamer. It then transfers them to either AWS S3 or an FTP server. By default, only the directory is configured, to the same location that Deep Learning Streamer is configured to output its capture files: /adlinkedge/capture/weldStream1. Any file that is detected in that directory will be uploaded automatically to your configured destination.
To configure the application to upload the files to the desired destination, you must add a new AWSS3 or FTPServer destination section to the config.
Once a destination is configured, the app will delete files from the local file system automatically after upload, to disable this, set DeleteFilesAfterUpload to false.
Configuring an AWS S3 bucket
- AWSS3
- CreateBuckets: Set this to true if you want Training Manager to automatically create an S3 bucket for you if it does not exist.
- Region: If you have set CreateBuckets to true, set this to a valid AWS region where you would like to create your bucket.
- Bucket: Enter the name of the bucket where you would like to store your files.
When you configure AWSS3, you must also provide your AWS credentials as environment variables. The AWS user that you configure must have permission to write to the bucket. More information on AWS credentials can be found in this article and information on S3 policies and permissions can be found in this article. Once you have the required credentials, go to the Docker tab of the Training Manager application and add your credentials as environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Configuring an FTP server
- FTPServer
- Host: Add the host URL or IP address here.
- Port: Specify the port, 21 for FTP and 22 for SFTP
- Username: Your FTP username
- Password: Your FTP password (optional)
- SFTP: Configure this if you want to use SFTP
- PublicKey: The path to your public key within the Training Manager container
- PrivateKey: The path to your private key within the Training Manager container
If you configure SFTP with a public and private key, be sure to add the keys to the Training Manager container in the Files tab.
Node-RED application
The Node-RED application does not need to be configured. By default, it comes complete with an example flow file that receives inference results from the default weldStream1 stream and displays the current weld status on a local web dashboard, accessible at http://<Device IP>:1880/ui. If the flow is subsequently changed by users, the flow can be exported and added to the application’s profile by adding the file to /adlinkedge/config/flows.json using the Files tab.
Exporting the Profile
Once the profile has been configured it can be exported from Profile Builder as a docker-compose bundle.
- On the intel-arc-welding-x86 profile page click Deploy
- On the window that appears, choose Download a docker-compose and click Next
Downloading the AI model and demo video file
ADLINK provides a zip archive containing the following files to support the deployment of the profile:
- models/weld_porosity_detection.* This is a pre-trained model designed to detect weld defects. The included label files provide label names that correspond to the three possible states supported by the model: no_weld, normal_weld, and porosity_weld (i.e. a weld defect).
- videos/weld.mp4 This is a 15-minute demo video that enables you to see how the solution would work in practice without the need for a uEye camera or arc welding system.
Click here to download the supplementary zip archive.
Running the Profile
After exporting your profile as a docker-compose bundle and downloading the supplementary zip archive in the above two steps, you must transfer the files to your edge device. Assuming a Ubuntu 18.04 LTS base operating system, this can be done using ssh. The following commands demonstrate this, replace user with your user name on the device, and edgedevice with the IP address or hostname of the device:
scp intel-arc-welding-x86.zip user@edgedevice:
scp intel-arc-welding-x86-resources.zip user@edgedevice:
After transferring the files, connect to the device Edge device via SSH and extract the files into the required directory structure:
ssh user@edgedevice
sudo apt install unzip
unzip intel-arc-welding-x86.zip -d intel-arc-welding-x86
unzip intel-arc-welding-x86-resources.zip -d intel-arc-welding-x86
You should end up with the below directory structure:
user@edgedevice:~$ tree intel-arc-welding-x86
intel-arc-welding-x86
├── deep-learning-streamer
│ └── adlinkedge
│ └── config
│ └── DeepLearningStreamer.xml
├── docker-compose.yaml
├── frame-streamer
│ └── adlinkedge
│ └── config
│ └── FrameStreamer.xml
├── models
│ ├── weld_porosity_detection.bin
│ ├── weld_porosity_detection.json
│ ├── weld_porosity_detection.labels
│ └── weld_porosity_detection.xml
├── node-red
│ └── adlinkedge
│ └── config
│ └── flows.json
├── training-manager
│ └── adlinkedge
│ └── config
│ └── TrainingManager.xml
└── videos
└── weld.mp4
14 directories, 10 files
To run the profile, first, install docker-compose, and then, from within the directory you extracted the files to, use docker-compose to bring up the profile:
sudo apt install docker-compose
cd intel-arc-welding-x86
sudo docker-compose up -d
To list running containers you can use the following command:
sudo docker ps
To check the logs for a running container you can use the following command:
sudo docker logs deep-learning-streamer
Once you are finished running the profile, you can bring it down with the following command:
sudo docker-compose down
Viewing the Inference Results
Viewing the Inference Results in VMLink
In addition to displaying video frames from frame streamer applications, VMLink automatically overlays any inference results associated with the stream on top of the images.
Video streams can be viewed within VMLink using the Streamer application.
Viewing the saved captures and inference data
With the default configuration, when capture is triggered in Deep Learning Streamer, the video and inference metadata files will be saved in the ./capture/weldStream1 directory within the extracted profile directory.
Viewing the Inference Results in Node-RED
An example Node-RED Dashboard is included with the inference profile and can be viewed by navigating to the following URL in a web browser:
http://<Device IP>:1880/ui
The dashboard shows the current weld status of weldStream1.
Node-RED Inference results post-processing
To reduce the chance of false positive results, the Node-RED flow performs some post-processing of inference results. The dashboard UI provides a few settings to configure this:
- Porosity frames: The number of porosity classifications in a row required to trigger porosity detection.
- Reset seconds: The number of seconds after the last porosity detection before resetting back to No Issues.
- Probability threshold: The minimum probability threshold required to accept a porosity classification as valid.
When the number of porosity classifications in a row required to trigger a porosity detection has been reached, the Node-RED flow will display Porosity Weld Detected on the dashboard. The DataRiver out node will then write to the PorosityEvent tag group shown below. The PorosityDetected tag will be set to true.
{
"name": "PorosityEvent",
"context": "com.adlinktech.weld",
"qosProfile": "event",
"version": "1.0.0",
"description": "A tag group to indicate porosity events for an arc welder.",
"tags": [
{
"name": "PorosityDetected",
"description": "True if porosity is detected, false if no porosity is detected.",
"kind": "BOOLEAN",
"unit": ""
}
]
}
A number of seconds after the last porosity detection, the Node-Red flow will display back to “No issues” on the dashboard. The DataRiver out node will then write to the PorosityEvent tag group with the PorosityDetected tag set to false.
Acting on Inference Results
When the Node-RED flow writes to the PorosityEvent tag group on the DataRiver, it provides an event that you may choose to act upon. One action that you may want to perform is to stop the arc welder when a weld defect is detected.
When a sample is received on the PorosityEvent tag group where the PorosityDetected tag is set to true; you can send a signal to the arc welder to stop via Modbus or OPC-UA.
The protocol choice will be dependent on the specifications of your arc welder. Depending on your protocol choice, the following sections describe how to configure either the Modbus or OPC-UA apps.
Modbus
Adding the Modbus app to your profile
First, you must add the Modbus app to the profile in Edge Profile Builder.
- From the intel-arc-welding-x86 profile page, click Add application.
- Search for aea-modbus and click on the application tile.
- On the aea-modbus application page, choose version 3.0.0 and click Add application.
- On the Adding aea-modbus panel that appears, ensure the intel-arc-welding-x86 profile is selected, enter the name modbus, and click Add.
Configuring the Modbus app
The following XML shows an example configuration of the Modbus app which will write the value of the PorosityDetected tag to a Modbus coil. For Further information on individual configuration settings, refer to the Modbus app documentation by clicking on the Documentation tab within Profile Builder.
<?xml version="1.0" encoding="UTF-8"?>
<Modbus xmlns="http://www.adlinktech.com/vortex-edge/0.9/Modbus">
<MODBUSConfiguration>
<Connection>
<TCP>
<IP>127.0.0.1</IP>
<Port>502</Port>
</TCP>
<SlaveID>1</SlaveID>
</Connection>
<Publication>
<FunctionCode>FC05 Write Single Coil</FunctionCode>
<StartAddress>0</StartAddress>
<Slices>
<Name>modbus.PorosityDetected</Name>
<Kind>UINT8</Kind>
<StartIndex>0</StartIndex>
<Expression>tagGroup.PorosityEvent.PorosityDetected</Expression>
</Slices>
</Publication>
</MODBUSConfiguration>
<DataRiverConfiguration>
<Thing>
<ContextID>com.adlinktech.modbus</ContextID>
<Class>
<Context>com.adlinktech.modbus</Context>
<Version>v1.0</Version>
<Input>
<TagGroupPath>file:///PorosityEventTagGroup.json</TagGroupPath>
<PublicationAlias>PorosityEvent</PublicationAlias>
</Input>
</Class>
</Thing>
</DataRiverConfiguration>
</Modbus>
Click View as XML on the Modbus app configuration page and paste in the above XML. Download the PorosityEventTagGroup.json file and add it to the root of the Modbus app using the Files tab.
Repeat the Exporting the Profile and Running the Profile steps to run your modified profile. You should now see the value of your Modbus coil change from 0 to 1 when weld defects are detected.
OPC-UA
Adding the OPC-UA Client app to your profile
First, you must add the OPC-UA Client app to the profile in Profile Builder.
- From the intel-arc-welding-x86 profile page, click Add application.
- Search for aea-opc-ua-client and click on the application tile.
- On the aea-opc-ua-client application page, choose version 1.1.0 and click Add application.
- On the Adding aea-opc-ua-client panel that appears, ensure the intel-arc-welding-x86 profile is selected, enter the name opc-ua-client, and click Add.
Configuring the OPC-UA Client app
The following XML shows an example configuration of the OPC-UA Client app which will write the value of the PorosityDetected tag to an OPC-UA node. For Further information on individual configuration settings, refer to the OPC-UA Client app documentation by clicking on the Documentation tab within Profile Builder.
<?xml version="1.0" encoding="UTF-8"?>
<OPCUAClient xmlns="http://www.adlinktech.com/vortex-edge/0.9/OPCUAClient">
<OPCUAConfiguration>
<ServerConnection>
<OPCUAServerEndpoint>opc.tcp://localhost:53530/OPCUA/Server</OPCUAServerEndpoint>
</ServerConnection>
<Publication>
<Alias>PorosityDetected</Alias>
<NodeNamespace>0</NodeNamespace>
<NodeIDString>PorosityDetected</NodeIDString>
</Publication>
</OPCUAConfiguration>
<DataRiverConfiguration>
<Thing>
<Class>
<Context>com.adlinktech.opcua</Context>
<Input>
<InputExpression>
<Name>PorosityEvent</Name>
<Context>com.adlinktech.weld</Context>
</InputExpression>
<Mapping>
<TagName>PorosityDetected</TagName>
<PublicationAlias>PorosityDetected</PublicationAlias>
</Mapping>
</Input>
</Class>
</Thing>
</DataRiverConfiguration>
</OPCUAClient>
Click View as XML on the OPC-UA Client app configuration page and paste in the above XML.
Repeat the Exporting the Profile and Running the Profile steps to run your modified profile. You should now see the value of your OPC-UA node change from false to true when weld defects are detected.