Using Model Confidence app to identify when a specific object has a low confidence level of recognition

Reading Time: 4 minutes

When operating AI models in the real world you want to be able to collect training data when the confidence level of a model running falls below a set threshold.

Within the vizi-ai-starter-kit there are a number of applications to start the process of configuring your Vizi-AI to gather training images automatically firstly we want to configure the ‘training-streamer’.

To get started open the ‘training-streamer’ app by clicking on the app name and the default app configuration displays. Complete the ‘logLevel’ and ‘contextId’ as required and click ‘+ Add New’ next to ‘Ftp’ section.

The following fields are displayed for completion:

After completing that section you need to enter the ‘StreamId’ from where the images are sourced. The streamID that the vizi-ai-starter-kit template is configured is called “demoStream”.

I have detailed below an example of some criteria that a user may wish to extract from the default configuration within the vizi-ai-starter-kit.   

This example defines that should an object be identified in the demoStream and the average probability given is returned as below the set threshold, the training streamer captures the image and sends it to the designated folder for the model to be retrained from.

These entries show that when the ‘demoStream’ is running and any the average model confidence is defined as having an ‘AverageProbability’ of less than ‘0.75’ it saves a file to your FTP site for retraining.

Once you have entered your criteria ensure that you save the amended configuration by clicking the ‘Save changes’ button at the top of the screen then ‘Close’ to return to the profile.

Now the ‘Training-Streamer’ has been configured you need to configure the ‘Model-Confidence’ app.

Click on the name of the ‘Model-confidence’ app to open it, you can start to configure the application from the default provided as per below.

Depending on what criteria was entered in the ‘Training-Streamer’ application, by default the ‘streamId’ within the vizi-ai-starter-kit is demoStream.

Based on the criteria entered in ‘Training-Streamer’ shown earlier I must enter the values in the ‘Streammetrics’ as shown below. 

You can see I have copied the ‘streamId’ that was in the ‘training-streamer’ and entered a preferred ‘timeWindow’ of 10 seconds, this calculates a confidence value based on the prediction probabilities across a sliding window over this time.

Once you are satisfied with your criteria ensure that you save the amended configuration by clicking the ‘Save Changes’ button at the top of the screen then ‘Close’ to return to the profile.

Now you need to deploy the profile to your Vizi-AI, to do this click the ‘Deploy’ button on the profile screen.

You are presented with three options to deploy your profile. Select Deploy directly to a device and click Next.

You will then be given the option to select the device that you want to deploy your profile directly to.

Select the device and click deploy.

Once you see the green Success notification at the bottom of the screen this means your profile has successfully been sent to the device, it may take a minute or two for the profile to be deployed, you can now click Close to exit the deployment option.

Once your profile is running and your stream can be monitored, the files will start to appear in the folder location like below when they fit the criteria given in ‘training-streamer’

Stay in touch

Sign up to our email list to be notified of the latest industry news