Dr. Daniel Lehmann

Convolutional Neural Network-based image classification models are the current state-of-the-art for solving image classification problems. However, obtaining and using such a model to solve a specific image classification problem presents several challenges in practice. To train the model, we need to find good hyperparameter values for training, such as initial model weights or learning rate. However, finding these values is usually a non-trivial process. Another problem is that the training data used for model training is often class-imbalanced in practice. This usually has a negative impact on model training. However, not only is it challenging to obtain a Convolutional Neural Network-based model, but also to use the model after model training. After training, the model might be applied to images that were drawn from a data distribution that is different from the data distribution the training data was drawn from. These images are typically referred to as out-of-distribution samples. Unfortunately, Convolutional Neural Network-based image classification models typically fail to predict the correct class for out-of-distribution samples without warning, which is problematic when such a model is used for safety-critical applications. In my work, I examined whether information from the layers of a Convolutional Neural Network-based image classification model (pixels and activations) can be used to address all of these issues. As a result, I suggest a method for initializing the model weights based on image patches, a method for balancing a class-imbalanced dataset based on layer activations, and a method for detecting out-of-distribution samples, which is also based on layer activations. To test the proposed methods, I conducted extensive experiments using different datasets. My experiments showed that layer information (pixels and activations) can indeed be used to address all of the aforementioned challenges when training and using Convolutional Neural Network-based image classification models.

An image classification model based on a Convolutional Neural Network architecture generally achieves a high classification performance over a wide range of image domains. However, the model is only able to obtain such a high performance on in-distribution samples. On out-of-distribution samples, in contrast, the performance of the model may be significantly decreased. To detect out-of-distribution samples, Papernot and McDaniel (2018) introduced a method named DkNN, which is based on calculating a sample credibility score by a nearest neighbor classification in feature space of the hidden layers of the model. However, a nearest neighbor classification is memory-intensive and slow at inference. To address these issues, Lehmann and Ebner (2022) suggested a method named LACA, which calculates the credibility score based on clustering instead of a nearest neighbor classification. Lehmann and Ebner (2022) showed that for out-of-distribution samples with respect to models trained on MNIST, SVHN, or CIFAR-10, LACA is significantly faster at inference compared to DkNN, while obtaining a similar performance. In this work, we conducted additional experiments to test LACA on more complex datasets (Imagenette, Imagewoof). Our experiments show that LACA is significantly faster at inference compared to DkNN also for these more complex datasets. Furthermore, LACA computes meaningful credibility scores, while DkNN fails on these datasets.

A convolutional neural network model is able to achieve high classification performance on test samples at inference, as long as those samples are drawn from the same distribution as the samples used for model training. However, if a test sample is drawn from a different distribution, the performance of the model decreases drastically. Such a sample is typically referred to as an out-of-distribution (OOD) sample. Papernot and McDaniel (2018) propose a method, called Deep k-Nearest Neighbors (DkNN), to detect OOD samples by a credibility score. However, DkNN are slow at inference as they are based on a kNN search. To address this problem, we propose a detection method that uses clustering instead of a kNN search. We conducted experiments with different types of OOD samples for models trained on either MNIST, SVHN, or CIFAR10. Our experiments show that our method is significantly faster than DkNN, while achieving similar performance.

Image classification problems are often class-imbalanced in practice. Such a class imbalance can negatively affect the classification performance of CNN models. A State-of-the-Art (SOTA) approach to address this issue is to randomly undersample the majority class. However, random undersampling can result in an information loss because the randomly selected samples may not come from all distinct groups of samples of the class (subclasses). In this paper, we examine an alternative undersampling approach. Our method undersamples a class by selecting samples from all subclasses of the class. To identify the subclasses, we investigated if clustering of the high-level features of CNN models is a suitable approach. We conducted experiments on 2 real-world datasets. Their results show that our approach can outperform a) models trained on the imbalanced dataset and b) models trained using several SOTA methods addressing the class imbalance.

Convolutional neural network (CNN) models are widely used for image classification. However, CNN models are vulnerable to out-of-distribution (OoD) samples. This vulnerability makes it difficult to use CNN models in safety-critical applications (e.g., autonomous driving, medical diagnostics). OoD samples occur either naturally or in an adversarial setting. Detecting OoD samples is an active area of research. Papernot and McDaniel (2018) have proposed a detection method based on applying a nearest neighbor (NN) search on the layer activations of the CNN. The result of the NN search is used to identify if a sample is in-distribution or OoD. However, a NN search is slow and memory-intensive at inference. We examine a more efficient alternative detection approach based on clustering. We have conducted experiments for CNN models trained on MNIST, SVHN, and CIFAR-10. In the experiments, we have tested our approach on naturally occurring OoD samples, and several kinds of adversarial examples. We have also compared different clustering strategies. Our results show that a clustering-based approach is suitable for detecting OoD samples. This approach is faster and more memory-efficient than a NN approach.

Before a neural network can be trained the network weights have to be initialized somehow. If a model is trained from scratch, current approaches for weight initialization are based on random values. In this work we examine another approach to initialize the weights of convolutional neural network models for image classification. Our approach relies on presetting the weights of convolutional layers based on information given in the training images. To initialize the weights of convolutional layers we use small patches extracted from the training images to preset the filters of the convolutional layers. Experiments conducted on the MNIST, CIFAR-10 and CIFAR-100 dataset show that using image patches for the network initialization performs similar to state-of-the-art initialization approaches. The advantage is that our approach is more robust with respect to the learning rate. When a suboptimal value for the learning rate is used for training, our approach performs slightly better than current approaches. As a result, information given in the training images seems to be useful for network initialization resulting in a more robust training process.