Model testing

The following code details the functions and classes that are available to test the performance of a trained model. The model can be used to classify images directly or calculate various metrics on a validation/test dataset. Multilabel image classification can also be performed using the library with a similar approach.

class decavision.model_testing.testing.ModelTester(model)

Class to use a trained image classification model on some images. Using this when working with a TPU disables eager execution.

Parameters

model (str) – path to trained model

classify_images(image_path, categories, plot=True)

Classify images located directly in a folder. Plots the images and the first three predictions.

Parameters

• image_path (str) – location of the images

• categories (list[str]) – list of potential categories that the model can return

• plot (bool) – plot or not the images, if False, only results are printed

confusion_matrix(path, normalize=None)

Compute and plot the confusion matrix resulting from predictions on a dataset of images. Images must be located in separate folders for each class.

Parameters

• path (str) – location of the images

• normalize ('true', 'pred', 'all' or None) – normalizes confusion matrix over the true (rows), predicted (columns) conditions or all the population. If None, confusion matrix will not be normalized.

evaluate(path)

Calculate the accuracy of the model on a dataset of images. The images must be in separate folders for each class.

Parameters

path (str) – location of the dataset

generate_classification_report(path)

Computes classification report resulting from predictions on a dataset of images and prints the results. Images must be located in separate folders for each class. The report shows average accuracy, precision, recall and f1-scores. Precision, recall and f1-scores are also computed for each class.

Parameters

path (str) – location of the images

plot_errors(path, num_pictures=9)

Plot images that were not classified correctly by the model. Images to test must be located in separate folders for each class, for example a validation dataset.

Parameters

• path (str) – location of the images

• num_pictures (int) – maximum number of errors to show

class decavision.model_testing.testing.ModelTesterMultilabel(model, categories)

Class to use a trained multilabel image classification model on some images. Using this when working with a TPU disables eager execution.

Parameters

• model (str) – path to trained model

• categories (list[str]) – list of potential categories that the model can return

classify_images(path, json_file, threshold=0.5, plot=False, save_img=False)

Classify images located directly in a folder. Plots and saves the images with the specified threshold.

Parameters

• path (str) – location of the images

• json_file (str) – path to json file containing image ids and their associated labels

• threshold (int) – threshold for prediction (default to 0.5)

• plot (bool) – plot or not the images, if False, only results are printed

• save_img (bool) – save classified images or not in a new folder. To save images, you will need to set plot=True, otherwise no images will be saved.

create_movie(classify_images, path='', threshold=0.5, json_file='', image_folder='')

Create a movie from classified images.

Parameters

• path (str) – location of the images

• classify_images (bool) – location of the classified images, set this to False if you have the images already saved

• threshold (int) – threshold for prediction (default to 0.5)

• json_file (str) – path to json file containing image ids and their associated labels

• image_folder (str) – if using already classified saved images, provide path to those images.

evaluate(path, json_file)

Calculate the f1-score of the model on a dataset of images. The images must be in single folder.

Parameters

• path (str) – location of the dataset

• json_file (str) – path to json file containing image ids and their associated labels

generate_metrics(path, json_file, threshold=0.5)

Computes classification report and confusion matrix resulting from predictions on a dataset of images and prints the results. Images must be located in a single folder.

Parameters

• path (str) – location of the images

• json_file (str) – path to json file containing image ids and their associated labels

• threshold (int) – threshold for prediction (default to 0.5)