Indoor surface classification for mobile robots

Convolutional layers

The most crucial part of CNN is the convolutional layer, which uses various convolution kernel sizes to extract the particular features of a given image. A set of feature maps can be obtained by applying a convolutional layer in a given image a few times. Assuming that F_i represents the CNN feature map for the ith layer, F_i is given as: (2)${\displaystyle F_i=\varphi \left(F_{i-1}{W}_i+b_i\right)}$ where F_i indicates the current network layer’s feature map, F_i−1 represents the previous layer’s convolution feature, W_i represents ith layer’s weight, b_i is the offset vector of the ith layer, and ϕ represents the rectified linear unit (ReLU) activation function.

Pooling layers

After applying a convolutional layer, pooling layers reduce the feature dimensionality, thus drastically minimizing the computational complexity. In addition, pooling layers effectively control the risk of over-fitting. The output feature of the hth local receptive field in the lth pooling layer can be calculated as follows: (3)${\displaystyle x_h^l=downsample\left(x_h^{l-1},s\right)}$ where downsample indicates the down-sampling function, $x_h^{l-1}$ represents the feature vector of the previous layer, and s denotes the pooling size.

Dense layers

Finally, a fully connected layer to perform the classification task collects all the features extracted by previous layers. The Softmax function generally performs class prediction with all the gathered features. The Softmax function can be expressed mathematically as: (4)${\displaystyle softmax{\left(z\right)}_j=e^{z_j}/\sum _{k=1}^K{e}^{z_k},\forall j\in \left\{1,\dots ,K\right\}}$ where K denotes the dimension of vector z.

InceptionV3

The InceptionV3 is a CNN based DL model that is used for image recognition tasks (Szegedy et al., 2016). It is an improved version of the main Inception V1 model. In the InceptionV3 model developed by Google, a group normalization layer and a fully connected layer have been added to make the network more efficient. This architecture has shown promising results on the ImageNet dataset and is widely used for image classification. The improved version aims to perform well even on applications with limited computational costs. For instance, a 3 × 3 convolution is divided into 3 × 1 and 1 × 3 convolutions, which allows for more effective extraction of specific details present in the image and reduces the numerical parameters of the model, resulting in shorter training times. The total depth of the model is 47 layers. Despite its more profound architecture than previous versions, InceptionV3, with the new factorization ideas, reduces the parameters without decreasing the network performance.

VGG16

VGG16 as a CNN-based model, was ranked among the top 5 models in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Russakovsky et al., 2015) competition on ImageNet dataset in 2014 with a 7.3% error rate (Simonyan & Zisserman, 2014). The architecture comprises 16 layers with approximately 130 million parameters, applying 3 × 3 filters. VGG16 focuses on 3 × 3 filtered convolution layers with a stride of 1 and a 2 × 2 filtered max-pool layer instead of many hyper-parameters. The architecture’s fundamental components are convolutional layers with varying depths, followed by three fully connected layers. The first and second dense layers contain 4,096 neurons, while the third has 1,000 neurons. Finally, the last layer is the softmax layer, where classification is performed.

VGG19

The overall concept of the VGG19 architecture is the same as VGG16, except for the adoption of three additional convolutional layers. In this case, the first 16 convolutional layers extract features, while the following three fully connected layers are used for classification. The feature extraction layers are divided into five groups, each followed by a max pooling layer. The input image size of the VGG19 model is the same as VGG16, which is 224 × 224 pixels.

ResNet50

The classification accuracy of deep CNN models increases correspondingly with the number of network layers. However, as the network depth increases, the model’s training and test error rate also increases. This phenomenon is referred to as the “vanishing gradient”. To overcome this problem, the Residual Network was introduced (He et al., 2016). ResNet50 deploys a technique called skip connections, allowing the network to link directly to the output by skipping several training layers. The main architecture of ResNet50 is inspired by the VGG architecture, consisting of five phases, each with an identity block with three convolutional layers and a convolutional block with three convolutional layers. A downsampling convolutional layer with a stride of 2 performs down-sampling. The network finalizes with fully connected layers.

Xception

As an inspired version of the Inception model, Xception architecture was introduced by Chollet (2017) in 2017. The architecture of the Xception model consists of deep and comprehensive convolutional layers which operate in a collateral manner. Thus, the feature extraction process of the network is realized with a total of 71 convolutional layers. In Xception, the convolutional layers are arranged into modules and encircled by linear residual connections, making it a stack of depthwise separate convolutional layers covered by residual connections. This makes the architecture very simple to describe and perform, unlike Inception V2 or V3, which are substantially more difficult to specify. The pre-trained model was trained on the ImageNet dataset on millions of images, offering high efficiency in image recognition.

InceptionResNetV2

The InceptionResNetV2 module is a combination of Inception and residual networks, showing promising results in image classification and object detection tasks (Szegedy et al., 2017). From a particular view, each InceptionResNet module can be demonstrated as a tiny CNN. These small CNNs, together with additional network layers like pooling and convolutions, compose the main architecture of the InceptionResNet network. The architecture of the InceptionResNet module consists of two main parts: shortcut connections and residual blocks. Input features are directly mapped to output features by the shortcut connection, so residual blocks estimate the residual function. Finally, the module output is composed of the corresponding map of input features as well as the output of the residual block.

MobileNetV2

MobileNetV2 model is utilized in many computer vision applications, but mainly it offers high robustness performance in object recognition and segmentation tasks. It increases the state-of-the-art performance of mobile models and computationally limited devices across various benchmarks, activities, and model sizes (Sandler et al., 2018). On the main architecture of MobileNetV2, bottleneck levels, inverse residual structures, and connections are presented. This methodology makes real-time classification possible on computationally limited devices or smartphones. More detailed information about the fundamental architecture of MobileNetV2 can be found in Sandler et al. (2018).

MobileNetV2-Modified

The fundamental MobileNetV2 architecture inspires our proposed modified MobileNetV2 model. Once the core convolutional architecture has extracted the input images’ features, the global average pooling layer was used to minimize the size of the extracted features. So, the classification task is done in the fully connected layer. The original MobileNetV2 model has a lower computational cost than other CNN-based models. Thus, this makes the MobileNetV2 model more effective in mobile robot-based applications. In this study, since the surface classification performance of the mobile robot was analyzed, it is more consistent to modify the MobileNetV2 model compared to other models in terms of computational cost. While the original MobileNetV2 model has an average weight of around 40 MB, which is relatively small, its architecture is changed to reduce the mobile robot’s computational cost and make it work more efficiently. To achieve this, we replace the last 11 layers before the fully connected layers of the original MobileNetV2 with a 2D-convolutional layer with 16 filters with a kernel size of 5x5, ReLu as activation, and the same padding. Thus, the number of blocks was reduced from 17 to 16, as shown in Figs. 4 and 5. The convolutional layer is followed by a Global Average Pooling layer and a Dropout layer with a value set to 0.2. Then, two dense layers are added, having a size of 2,096 and 3, respectively. Softmax is used for activation to classify the three types of surfaces correctly. The size of the extracted features is minimized using the Global Average Pooling layer, and overfitting was prevented by adding a new Dropout layer to the modified architecture. Apart from removing blocks, adding dropouts, building a convolution layer, we made a significant change to the proposed model by making the last 23 layers trainable (Fig. 5), whereas in the original MobileNetV2, all layers are frozen except for the fully connected and global average pooling layers. In the MobileNetV2 model, the frozen layers are trained on the ImageNet dataset, while the trainable layers are trained with a custom dataset. By doing so, we made 23 layers of the modified model trainable. This indicates that if we did not make any modifications to the architecture, the number of trainable parameters in the modified model would be higher than that of the original model. However, we removed the last 11 layers of the model and downsized the filter size of the last convolutional layer in the architecture. Therefore, we simplified the model, resulting in fewer trainable parameters, less training time and smaller model weight size. Note that the modification of the architecture was finalized after several rounds of fine-tunings.

Stochastic Gradient Descent

Stochastic gradient descent (SGD) (Robbins & Monro, 1951) and its variants are frequently used in deep learning. SGD is different from the classic vanilla gradient descent technique in that it computes the gradient of the objective function. It updates the parameters on subsets of the training data instead of the complete training set. The main goal of SGD is to minimize computational costs. It is worth noting that gradient descent is a computationally intensive process by itself, especially when there is a large training set. The parameter update of SGD is done as follows (Zhao et al., 2019): (5)${\displaystyle {\theta }_t={\theta }_{t-1}-\eta \cdot {\nabla }_{\theta }J\left({\theta }_{t-1}\right)}$ where θ_t ∈ R^d is the parameter vector during iteration t, the learning rate is represented by η, J(θ_t−1) is the loss function defined by the model’s parameters θ_t at the t-th iteration, and ∇_θJ(θ_t−1) = ∂J(θ_t−1)/∂θ represents the gradient of the loss function to parameters at the (t − 1)th iteration.

Adaptive Moment Estimation

The advantages of Adagrad and RMSprop optimizers are combined in adaptive movement estimation (Adam), which calculates the learning rates adaptively for various variables. It has a low memory demand and a high computational efficiency and calculates the learning rates for each parameter. Momentum is directly integrated with Adam to estimate the gradient’s first-order moment (Soydaner, 2020). To account for the initialization at the origin, Adam applies bias corrections to the estimations of the first-order and second-order moments (Kingma & Ba, 2014). To estimate the moments, Adam employs the exponential moving average, which is computed on the gradient evaluation with respect to the current mini-batch. The exponential decay rates of these moving averages are controlled by the hyperparameters β₁ and β₂, which are usually set close to 1. β₁ and β₂ are typically set to values close to 1. The estimate of the gradient’s mean, m_t, is calculated as: (6)${\displaystyle m_t={\beta }_1{m}_{t-1}+\left(1-{\beta }_1\right)g_t}$ where g_t is the gradient on the current mini-batch at step t. Subsequently, the estimate of the gradient’s uncentered variance, v_t, is calculated as: (7)${\displaystyle v_t={\beta }_2{v}_{t-1}+\left(1-{\beta }_2\right)g_t^2}$

In the first stage, m_t and v_t are smoothly biased to starting value. Therefore, bias correction must be calculated for both the first and second moments, so the following steps are performed: (8)${\displaystyle {\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t}}$ (9)${\displaystyle {\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}}$

Finally, the parameters are updated as follows: (10)${\displaystyle {\theta }_t={\theta }_{t-1}-\frac{\eta }{\sqrt{{\hat{v}}_{t-1}}+ϵ}{\hat{m}}_{t-1}}$ where η is the learning rate and ϵ is a small value to avoid division by zero.

Root Mean Square Propagation

The root mean square propagation (RMSProp) algorithm was proposed by Geoffrey Hinton (Hinton, Srivastava & Swersky, 2014). RMSProp optimizer is similar to the gradient descent algorithm with momentum. By employing a moving average of the squared gradient to normalize the gradient by considering the size of recent gradient descents, RMSProp attempts to address the drastically decreasing the learning rates of Adagrad. Therefore, the algorithm progresses horizontally, with larger steps converging faster as the learning rate increases (Mukkamala & Hein, 2017). The moving average of the squared gradient is maintained as follows: (11)${\displaystyle {\theta }_t={\theta }_{t-1}-\frac{\eta }{\sqrt{E{\left[g^2\right]}_{t-1}+ϵ}}{g}_{t-1}}$

Adagrad

Adagrad is an optimization technique that modifies the learning rates of the model parameters on an individual basis (Duchi, Hazan & Singer, 2011). The learning rate of the parameters with the biggest partial derivative of the loss decreases quickly, whereas the learning rate of the parameters with smaller partial derivatives decreases more slowly (Goodfellow, Bengio & Courville, 2016). This is accomplished by using all the previous squared values of the gradient. The update for each parameter θ_i in every iteration t is expressed as follows: (12)${\displaystyle {\theta }_{t{,}_i}={\theta }_{t-1}{,}_i-\frac{\eta }{\sqrt{G_{t-1{,}_{ii}}+ϵ}}\cdot g_{t-1{,}_i}}$

Here, ϵ is an expression that prevents division by zero, and the objective function’s gradient with respect to the parameter θ_i at iteration t − 1 is denoted with g_t−1: (13)${\displaystyle g_{t-1{,}_i}={\nabla }_{\theta }J\left({\theta }_i\right),}$ where g_t−1, _ii is a diagonal matrix where every diagonal element i, i is the sum of gradients squares in respect to θ_i up to iteration t − 1. Element-wise vector multiplication between G_t−1 and g_t−1 is represented with ⊙, and vectorization update is done as follows: (14)${\displaystyle {\theta }_t={\theta }_{t-1}-\frac{\eta }{\sqrt{G_{t-1}+ϵ}}\odot g_{t-1}}$

AdaDelta

The primary goal of the AdaDelta method is to address the two main issues with AdaGrad: the constant decay of learning rates during training and the demand for manually selecting global learning rates. To achieve this, AdaDelta limits the past gradient window to a fixed size rather than adding up the sum of all squared gradients over time (Zeiler, 2012). The update of AdaDelta is as follows: (15)${\displaystyle {\theta }_t={\theta }_{t-1}-\frac{RMS{\left[\Delta \theta \right]}_{t-2}}{RMS{\left[g\right]}_{t-1}}{g}_{t-1}}$ where RMS[g]_t−1 represents the RMS error criterion of the gradient [g]_t−1: (16)${\displaystyle RMS{\left[g\right]}_{t-1}=\sqrt{E{\left[g^2\right]}_{t-1}+ϵ}}$ and the running average at the t − 1th iteration is represented with E[g²]_t−1, which is calculated using the previous average and the exist gradient: (17)${\displaystyle E{\left[g^2\right]}_t=\gamma E{\left[g^2\right]}_{t-1}+\left(1-\gamma \right)g_{t-1}^2}$ where γ denotes a fraction that is identical as the momentum term. Following this, RMS[Δθ]_t represents the error criterion of Δθ², and the running average E[Δθ²]_t of Δθ_t is acquired by: (18)${\displaystyle E{\left[\Delta {\theta }^2\right]}_t=\gamma E{\left[\Delta {\theta }^2\right]}_{t-1}+\left(1-\gamma \right)\Delta {\theta }_{t-1}^2}$

Adamax

Adamax is an extension of Adam. First, Adamax calculates the gradients at time step t with respect to the stochastic objective. Then, it calculates an exponentially weighted infinity norm along with a biased first-moment estimate to update model parameters (Kingma & Ba, 2014). Adamax’s parameter update rule is as follows: (19)${\displaystyle u_t=max\left({\beta }_2{v}_{t-1},\left|g_t\right|\right)}$ (20)${\displaystyle {\theta }_{t+1}={\theta }_t-\frac{\eta }{u_t}{\hat{m}}_t}$

Nadam

Nadam uses the accelerated gradient of Nesterov to modify Adam’s momentum component. Thus, the goal of Nadam is to increase the speed of convergence of the models (Dozat, 2016). Similar to the Adam algorithm, the first and second-moment variables are updated after computing the gradients. Following this, corrected moments are calculated so that the parameters can be updated. The parameter updates are expressed as: (21)${\displaystyle {\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{{\hat{v}}_t}+ϵ}\left({\beta }_1{\hat{m}}_t+\frac{\left(1-{\beta }_1\right)g_t}{1-{\beta }^{t_1}}\right)}$