Self-supervised recurrent depth estimation with attention mechanisms

Supervised depth estimation

The idea of supervised depth estimation implies direct learning of a hidden mapping between the input image and its depth. Usually, this approach includes regression-based methods because of the continuous nature of depth maps.

One of the earliest applications of deep learning to this problem (Eigen, Puhrsch & Fergus, 2014) introduced a network that consists of two components: one estimates the global scene structure, and the other refines the results using local information. In the stereo-based approach, Xie, Girshick & Farhadi (2016) designed a convolution neural network that generates a corresponding right view of a stereo pair by combining information from multiple levels. Disparity distribution for pixels is optimized with supervised photometric loss. Cao, Wu & Shen (2016) reformulated the supervised depth estimation task as a pixel-wise classification problem that was solved with a fully convolutional deep residual network.

Supervised methods mostly rely on convolutional architectures. One important difference between methods is that they tend to utilize a large variety of objective functions that define the difference between the ground truth depth map and its prediction. The most straightforward approach is to use L₁ or L₂ distances. Reverse Huber loss (Laina et al., 2016b) is a popular example of a combination of these two distances. To address the scale problem, some studies focused on the loss as a distance between predicted and real depth in log space (Eigen, Puhrsch & Fergus, 2014; Eigen & Fergus, 2014). Yin et al. (2019) argued that most metrics neglect geometric constraints in the three-dimensional space, so they developed a loss that enforces geometrical constraints. Another alternative to pixel-wise metrics is perceptual loss (Johnson, Alahi & Li, 2016), which was successfully used in many depth-related tasks (Makarov, Aliev & Gerasimova, 2017; Makarov et al., 2017; Makarov, Korinevskaya & Aliev, 2018a; Makarov & Korinevskaya, 2019; Makarov, Korinevskaya & Aliev, 2018b; Makarov, Korinevskaya & Aliev, 2018c; Makarov et al., 2019; Maslov & Makarov, 2021). To reduce blurring of estimated depths, a new model with fusion mechanisms was proposed by Hu et al. (2019). It consists of a module that can fuse multi-scale information produced by an encoder, a refine module to fuse decoder outputs and multi-scale fusion modules. In their objective function, the authors combined the difference in gradients and surface normals with balanced Euclidean loss in log space.

State of the art performance in the supervised setting has been achieved by Fu et al. (2018), who proposed the SID policy and ordinal regression loss. Despite the excellent performance of the model, its main disadvantage is a huge number of parameters that leads not only to long training time but also to large inference times in practice. Therefore, a challenge arises to build a lightweight model that is able to process frame sequences in real-time with reasonable quality. Recently, Maslov & Makarov (2020a) proposed an architecture that appears to be suitable for real-time robotic applications. The model utilizes temporal information from image sequences with the help of a convolutional gated recurrent unit (ConvGRU) and convolutional long short-term memory (ConvLSTM). The authors showed that these recurrent blocks can lead to improvements in depth estimation accuracy.

Self-supervised depth estimation

As deep learning models for computer vision become larger and more expressive, the data requirements for training them rapidly grow as well. Creating a labeled dataset for depth estimation by hand or with special hardware is extremely laborious and expensive. One solution would be to use synthetic data (Nikolenko, 2021) that has indeed been extensively used for depth estimation. An alternative approach would be to learn from unlabeled monocular sequences or stereo image pairs, as has been done, e.g., in Garg, G & Reid (2016) and Godard, Aodha & Brostow (2016). This approach does not utilize ground truth depth maps. One of the first models to perform self-supervised depth estimation (Garg, G & Reid, 2016) consists of a convolutional neural network that estimates depth maps from paired stereo images. A predicted depth map is used in the geometrical transformation of the left image into the synthesized right image or vice versa. The loss function here consists of the difference between the synthesized and real images, defined as the structural similarity index measure (Wang et al., 2004), and depth regularization terms.

Further studies mostly focused on generalizations of this approach in the monocular setting. Monocular video-based methods usually utilize consecutive frames to simultaneously learn the depth and pose. Given a target image and adjacent frames as input, a model produces the depth for the target image and estimates relative poses between the target image and nearby images. Next, the target view is reconstructed with the predicted depth map and relative poses. The main difference between stereo-based and monocular video-based approaches is that poses for stereo data are known and fixed, while in the monocular setting, they also need to be predicted.

Most state-of-the-art methods in self-supervised depth estimation are based on encoder–decoder architectures. Next, we highlight a few such models that are the most relevant for our study. Monodepth2 Godard et al. (2019) has been probably the most influential approach in self-supervised depth estimation because of its simplicity. Its architecture consists of a depth net and a pose net; both nets consist of a ResNet encoder and a decoder. The main contribution of Monodepth2 is a set of improvements that together lead to superior results in depth estimation. The authors introduced the minimum reprojection loss as a substitute for the average reprojection loss and showed that the new loss is more effective in handling occlusions. In addition, Godard et al. (2019) proposed a multi-scale pyramidal upsampling method and auto-masking loss that reduced visual inconsistency. As a result, the model showed high-quality results on the KITTI benchmark, and since then has become a starting point for many researchers. In this work, we mostly follow Monodepth2 architecture and propose a set of additional components that can be used to extract information from previous frames.

Guizilini et al. (2020a) recently proposed a new architecture called PackNet-SfM. Its key feature is the design of packing and unpacking blocks that are considered to preserve the 3D geometrical structure of depth maps. Moreover, the authors presented a new velocity supervision loss that solves the problem of scale ambiguity. Depth scale ambiguity implies that predicted depth maps are usually scaled with predefined constants. In PackNet, camera velocity is estimated and used to compute the loss function that helps reduce the problem with scaling. PackNet has achieved excellent performance on KITTI, but it utilizes heavy 3D convolutions, resulting in a long processing time.

In this work, we intend to investigate the effect of temporal information in depth estimation in the self-supervised setting. Despite the fact that most existing models do not focus on capturing temporal knowledge, there is a number of novel models that try to follow a multi-frame or sequential approach in depth estimation (Watson et al., 2021; Hur & Roth, 2021; Kuznietsov, Proesmans & Van Gool, 2021). In particular, Watson et al. (2021) proposed to utilize nearby frames if they are available at test time and designed a model called ManyDepth. Its key feature is a cost volume block that helps overcome the scale ambiguity that arises from self-supervised training. Experiments on KITTI and Cityscapes show that ManyDepth outperforms most existing self-supervised approaches. Hur & Roth (2021) focused on the scene flow task where 3D scene flow and depth are estimated jointly in a self-supervised setting. They propose a multi-frame model that uses triplets of frames and recurrent blocks, and it shows state-of-the-art performance for the scene flow task.

Problem formulation

In the self-supervised monocular setting, we operate with frame sequences extracted from videos. Given these consecutive frames, we predict the target view from the viewpoint of another image. In order to synthesize the target image, we use a hidden variable, the depth map, and estimated relative pose between the target image and nearby view. The overall self-supervised pipeline is illustrated in Fig. 1.

Formally, the problem can be described as follows. Suppose that we are given N adjacent images that can be considered as a frame sequence {I₁, …, I_N}. Usually, one image I_t is chosen as the target image, while other frames $S={\left\{I_s\right\}}_{s=1,s\ne t}^N$ are considered as context. This sequence is an input for our model, which consists of two parts:

• the depth network (Depth Net in Fig. 1) f_D:I → D that predicts scale-ambiguous depth D = f_D(I(p)) for every pixel p in the input image I; using this depth network, we estimate the depth for the target image;

• the pose network (Pose Net in Fig. 1) f_P:(I_t, I_s) → P_t→s that estimates a set of 6-DoF relative poses P_t→s,  I_s ∈ S between the target image I_t and nearby context frames; for example, if N = 3 then I_t = I₂ and I_s = I_i, where i ∈ {1, 3}.

Given a pair (I_s, I_t) consisting of the target image and source image, we can inversely warp I_t to the source frame I_s. In order to do this view synthesis, we need to know an estimated depth map D_t for the target image I_t and the transformation P_t→s from I_t to I_s. For a given pixel coordinate p_t from I_t which is co-visible in I_s, its corresponding pixel coordinate p_s in I_s can be defined by the inverse warping transformation as follows: (1)${\displaystyle p_s\sim K_s\left[P_{t\to s}\right]D_t\left(p_t\right)K_t^{-1}{p}_t,}$

where ∼ denotes equality in the homogeneous coordinates (Shen et al., 2019), and K_s and K_t are intrinsic matrices that are known and fixed. D_t(p_t) is the depth value for a pixel p_t, and [P_t→s] is the transformation matrix. Thus, given the depth map and transformation matrix, we can construct a matching function between pixels of the source and target images. This approach can be considered as a geometric projection relationship, and it helps us synthesize views. The difference between a real image and a synthesized one is a key component in the loss function described below.

Objective function

In the objective function, we follow Zhou et al. (2017): the overall self-supervised loss function consists of the photometric reprojection error L_p and depth regularization term L_s.

Let I_t be a ground truth target image, and let I_s→t be the target image synthesized from the source image I_s. Then the basic photometric reprojection error L_p is defined as (2)${\displaystyle L_p=\sum _{I_s\in S}\mathrm{pe}\left(I_t,I_{s\to t}\right),}$ where the reprojection error is defined as the weighted combination of the structural similarity index measure (SSIM; see Wang et al. (2004)) and L₁ pixelwise difference: (3)${\displaystyle \mathrm{pe}\left(I_a,I_b\right)=\alpha \frac{\left(1-SSIM\left(I_a,I_b\right)\right)}{2}+\left(1-\alpha \right)\parallel I_a-I_b{\parallel }_1.}$ Following Godard et al. (2019), we set α = 0.85.

The depth regularization term, first introduced by Godard, Aodha & Brostow (2016), helps to regularize the depth in texture-less low-image gradient regions and is defined as (4)${\displaystyle L_s\left({\hat{D}}_t\right)=\left|{\delta }_x{\hat{D}}_t\right|e^{-\left|{\delta }_x{I}_t\right|}+\left|{\delta }_y{\hat{D}}_t\right|e^{-\left|{\delta }_y{I}_t\right|}.}$

As we have mentioned above, we calculate the reprojection loss for a set of source images I_s. A common strategy used to compute the final loss is to average the reprojection error from multiple sources, i.e., to average the expression in Eq. (2). However, this approach may cause problems with pixels that may be not visible in some source images. Such pixels represent a common problem in the ego-motion study because there is a high probability of encountering objects blocking each other, especially in outdoor scenes. Several works have shown that this problem can be overcome by using masking (Mahjourian, Wicke & Angelova, 2018; Vijayanarasimhan et al., 2017); however, they do not take into account the recovery of occluded regions. This phenomenon has been carefully studied by Godard et al. (2019), and a simple solution was proposed: the authors replaced averaging with minimum. As a result, the total photometric loss is defined as (5)${\displaystyle L_p=\underset{I_s\in S}{min}\mathrm{pe}\left(I_t,I_{s\to t}\right).}$

Multi-scale image reconstruction is a common technique in computer vision tasks used to prevent the objective function getting stuck in local minima. Following Godard et al. (2019), we also use multi-scaling in our model. More formally, the photometric loss is defined not just as on the final decoder output but as a combination of loss terms on every scale of the decoder. In this work, we consider 4 scales. Godard et al. (2019) revealed that this approach may result in “holes” in low-texture parts of the image, so Godard et al. (2019) provided a modification for the multi-scaling approach. They upsample depth maps at low scales of the decoder and then compute the photometric loss with these upscaled maps, so that decoder outputs in the loss computation are upscaled to the original image resolution. This approach has proven to be very effective in the self-supervised multi-scale depth estimation pipeline.

Self-supervised depth estimation mostly relies on ego-motion. Nevertheless, sometimes motion assumptions may be violated when the camera is in a stationary position but objects move in the scene. As a result, we may have stationary pixels that need to be processed by the model. To overcome these issues, some approaches have been using per-pixel masking (Zhou et al., 2017; Vijayanarasimhan et al., 2017; Luo et al., 2018). The mask is usually incorporated into the loss function, and it can be either predictive, as in Zhou et al. (2017), or computed by some direct algorithm. In this work, we use the auto-masking method proposed in Godard et al. (2019). The idea is to track pixels that remain the same between nearby frames, which implies a static position of the camera. The mask is binary, μ ∈ {0, 1}, where the value 0 corresponds to pixels where the reprojection error of the synthesized image I_s→t is lower than the error of the original unwrapped image I_s: (6)${\displaystyle \mu =\left[\underset{I_s\in S}{min}\mathrm{pe}\left(I_t,I_{s\to t}\right)<\underset{I_s\in S}{min}\mathrm{pe}\left(I_t,I_s\right)\right].}$

Recurrent block ConvGRU

Recurrent neural networks (RNNs) have been widely adopted in the field of computer vision. However, initial RNN models appeared to face difficult training problems, such as exploding or vanishing gradient effects (Pascanu, Mikolov & Bengio, 2012). A common way to overcome the vanishing gradient problem is to use long short-term memory units, LSTM (Hochreiter & Schmidhuber, 1997; Gers, Schmidhuber & Cummins, 2000). LSTMs have become one of the most popular approaches in sequence-to-sequence settings, especially in natural language processing (NLP) tasks, including speech recognition (Graves, Mohamed & Hinton, 2013) and machine translation (Parmar & Devi, 2019). By definition, an LSTM block operates with one-dimensional inputs and is not naturally suited to image processing. Nevertheless, Shi et al. (2015) recently proposed a new architecture called ConvLSTM that extends the basic LSTM construction to two-dimensional inputs.

Over the past few years, some studies have used an alternative recurrent mechanism called ConvGRU (Zhang et al., 2019). It was first introduced for the video representation task (Ballas et al., 2015) and has proven to be quite effective in subsequent studies (Siam et al., 2016; Maslov & Makarov, 2020a). ConvGRU has a simpler architecture than ConvLSTM while preserving the same gated structure. In this work, we use ConvGRU cells since it has fewer parameters, and it has been proven to be effective in capturing temporal information for online depth estimation (Maslov & Makarov, 2020a).

Next, we describe the ConvGRU cell as a particular type of RNN network applied to a sequence of 2D objects with an arbitrary length. ConvGRU can be viewed as a modification of the original GRU cell (Cho et al., 2014). The main difference between these GRU versions is that ConvGRU utilized convolution operations instead of fully connected units used in the original GRU. Each recurrent unit of the ConvGRU cell is designed to capture dependencies on different time scales. Moreover, due to convolution operation, these recurrent units share their parameters across different spatial locations in the input (Ballas et al., 2015). The hidden state h_t of the ConvGRU can be described by the following equations:

(8)${\displaystyle z_t=\sigma \left(W_{xz}\ast x_t+W_{hz}\ast h_{t-1}+b_z\right),}$ (9)${\displaystyle r_t=\sigma \left(W_{xr}\ast x_t+W_{hr}\ast h_{t-1}+b_r\right),}$ (10)${\displaystyle {\tilde{h}}_t=tanh\left(W_x\ast x_t+W_h\left(r_t\circ h_{t-1}\right)+b_h\right),}$ (11)${\displaystyle h_t=\left(1-z_t\right)\circ h_{t-1}+z_t\circ {\tilde{h}}_t,}$

where ∗ denotes convolution, weight matrices W_xz, W_hz, W_xr, W_hr, W_x, W_h are 2D-convolutional kernels, and b_z, b_r, b_h are the corresponding bias terms. In order to preserve the same spatial size of hidden representations, zero-padding is used in recurrent convolutions. The ConvGRU architecture is illustrated in Fig. 2.

Our main idea of using convGRU unit is similar to Maslov & Makarov (2020a), in which authors used combinations of convolutions and recurrent blocks for temporal information aggregation for supervised depth estimation. We believe that in self-supervised setting temporal consistency is even more essential for robust depth estimation in both, single image and sequential settings. The key component to it is properly chosen self-attention mechanism described next.

Self-attention

Convolutional neural networks are widely used in many neural network architectures, especially in computer vision. However, a single convolutional layer can only capture dependencies in a small neighborhood. Therefore, alternative approaches based on self-attention are also starting to arise (Ramachandran et al., 2019). Self-attention can be considered as an independent layer that can be a good substitute for convolutions. Experiments by Ramachandran et al. (2019) show that simply replacing spatial convolutions with self-attention in a ResNet-type architecture leads to higher accuracy in ImageNet classification and COCO object detection. In this work, we use self-attention layers to fuse temporal features.

Next, we describe the self-attention layer used in our architecture in detail; our exposition directly follows the original paper by Ramachandran et al. (2019). Similar to a convolution, given a pixel x_ij ∈ ℝ^d_in, we first take a neighborhood N_k(i, j) centered on this pixel that extends for k pixels around x_ij. This region is called a memory block.

One-headed attention is calculated as follows: (12)${\displaystyle y_{ij}=\sum _{a,b\in N_k\left(i,j\right)}{\mathrm{softmax}}_{ab}\left(q_{ij}^T{k}_{ab}\right)v_{ab},}$

where y_ij ∈ ℝ^d_out, queries q_ij = W_Qx_ij, keys k_ab = W_Kx_ab, values v_ab = W_Vx_ab.W_Q, W_K, W_V ∈ℝ^d_out×d_in are linear transforms. Similar to convolutions, self-attention aggregates spatial information over a neighborhood region but aggregation is a convex combination of value vectors with weights defined by the softmax operator. This procedure repeats for every pair of indices (i, j). Moreover, it is possible to use multiple attention heads, splitting the pixels x_ij depthwise into N groups $x_{ij}^{\left(n\right)}\in {\mathbb{R}}^{d_{in}/N}$, computing the output for each group separately, and then concatenating these outputs.

Next, to aggregate information from different resolutions, frames, and components, we need efficient fusion model taking the best out of convolutional and attention aggregation mechanisms.

Fusion

Fusion is a technique commonly used to aggregate multimodal features. Multimodality implies that objects have different semantic characteristics. Traditional fusion methods include concatenation or simple averaging. Numerous studies employ fusion for processing multimodal inputs, and many more complex fusion methods have been proposed over the last decade.

In this work, to fuse temporal features we use residual convolutional units as introduced by Lin et al. (2016) and their modification that we call residual self-attention units. A residual convolution unit itself is a simplified version of the convolution unit in the original ResNet (He et al., 2015). It consists of two spatial convolutions with ReLU activations and batch normalization. It also has a skip connection between the input and the output of convolution layers. In this work, we propose a modification of the residual convolution unit: we replace the spatial convolutional layer with a self-attention layer. We believe that this modification helps capture dependencies from different neighborhoods of the input’s representation, potentially resulting in more accurate predictions. Figure 3 shows a schematic overview of the residual self-attention unit (Fig. 3B) and residual convolutional unit (Fig. 3C).

Residual convolutional and self-attention units are used to fuse ConvGRU outputs and hidden states. Fusion modules progressively combine and upscale feature maps in order to generate a fine-grained prediction. Figure 3A presents a schematic overview of our Fusion block.

Finally, we are ready to describe overall model.

Recurrent self-supervised depth estimation

In this section, we describe the depth network pipeline and illustrate the role of recurrent blocks and fusion units in it. For this study, we have attempted to investigate and propose a modification to existing pipelines that would utilize temporal information across frame sequences. To examine the impact of temporal knowledge, we tested different modifications, and in this section, we describe them in more detail. We have experimented with depth network components, while the pose network remains the same as in previous models (Godard et al., 2019; Guizilini et al., 2020a; Zhou et al., 2017). The pose network consists of the ResNet18 encoder and pose decoder that is presented in Fig. 4.

The key feature of the proposed depth model is the use of two types of temporal information; it is schematically illustrated in Fig. 5. The first type of temporal information is the content of previous frames transferred via a recurrent block, i.e., via the hidden state. The second type of temporal knowledge comes from the dependencies between different scales of decoder outputs (indicated by green arrows in Fig. 5). The architecture design is inspired by Sun et al. (2021) who used similar types of temporal knowledge and connections between them for the real-time 3D scene reconstruction task. In Sun et al. (2021), the recurrent part of the model is based on 3D convolutions, and in this work we simplify their approach and adapt it to our task.

The depth model consists of three consecutive parts. The first part is an encoder–decoder block with skip connections, where the encoder is a ResNet18 and the decoder is a sequence of upsampling convolutional blocks as shown in Fig. 4. The decoder produces outputs on four different scales. We consider two types of decoder outputs: single-channel outputs and multi-channel outputs with n ∈ {16, 32, 64, 128} channels. In the decoder architecture, multi-channel outputs are usually mapped to single-channel ones via a convolutional layer, as in Godard et al. (2019). However, in this work we do not complete the depth prediction process at this stage but rather continue to process internal feature representations produced in the decoder. These representations are fed into the second part of our model, recurrent ConvGRU blocks. ConvGRU on a particular scale takes as input a concatenation of the depth output at this scale and an upscaled representation from the previous scale (except for, obviously, the first scale where ConvGRU receives only the depth output). The third part of the model is the Fusion block that helps aggregate temporal information. The head of the model consists of simple convolution blocks that produce disparity for a target image.

Next, we give a formal description of the full depth estimation pipeline for a given image. Let X_t be an input image, and let $\left(H_t^1,H_t^2,H_t^3,H_t^4\right)$ be the hidden states of the recurrent block on four scales produced from previous frames. The encoder–decoder network produces outputs on four scales s ∈ {1, 2, 3, 4}. We denote by $F_t^s$ the encoder–decoder output on scale s at time t: (13)${\displaystyle \left(F_t^1,F_t^2,F_t^3,F_t^4\right)=f_{Dec}\left(f_{Enc}\left(X_t\right)\right).}$

Then, we feed decoder outputs $F_t^s$ and upscaled maps from previous scales into ConvGRU blocks: (14)${\displaystyle G_t^s=f_{ConvGRU}^s\left(concat\left(F_t^s,U_t^{s-1}\right)\right),s\in \left\{2,3,4\right\},}$ (15)${\displaystyle G_t^s=f_{ConvGRU}^s\left(F_t^s\right),s=1.}$

Next, we fuse ConvGRU outputs and hidden states from the previous timestamp: (16)${\displaystyle Z_t^s=f_{Fusion}^s\left(G_t^s,H_t^s\right).}$

Fusion outputs then go to an upscaling block, resulting in $U_t^s$, and to the convolution disparity block that produces final depth map on this scale:

(17)${\displaystyle U_t^s=f_{Upscale}^s\left(Z_t^s\right),s\in \left\{0,1,2\right\},}$ (18)${\displaystyle {\hat{D}}_t^s=f_{ConvDisp}^s\left(Z_t^s\right),s\in \left\{0,1,2,3\right\}.}$

For upscaling, we use bilinear interpolation while working with single-channel decoder outputs and the PixelShuffle operation (Shi et al., 2016) while working with multi-channel decoder outputs.

KITTI Dataset

In order to compare the results with previously proposed monocular self-supervised methods, we evaluate our models on the KITTI 2015 dataset (Menze & Geiger, 2015). The data consists of outdoor driving scenes which usually include up to 15 cars and 30 pedestrians. Accurate ground truth depth maps that are needed for further evaluation are also provided.

We use the data split proposed by Eigen & Fergus (2014), with monocular frame sequences. We use the preprocessing technique suggested by Zhou et al. (2017) that removes static frames. In order to train the model on sequences, we randomly sample consecutive frames from available scenes. As a result, the training set contains 2685 monocular samples, and the validation set contains 176, where a sample is a sequence of frames. The length of the sequence is set to 10. Following Godard et al. (2019), we use the same camera intrinsics for all images, setting the principal point of the camera to the center of the image and focal length to the average of all focal lengths in KITTI. Training on image sequences is a challenging task for GPU memory limits. Therefore, we had to resize all input images to resolution 194 × 640. The same resizing technique was used in previous works (Godard et al., 2019; Guizilini et al., 2020a; Zhou et al., 2017), so comparison with existing models appears to be fair.

Training details

Working with recurrent neural networks introduces certain complications in the training strategy. We need to choose how to initialize hidden states and how to train the network. The approach to initializing hidden states with zeros is commonly used in many sequence-to-sequence natural language processing tasks, including machine translation (Parmar & Devi, 2019). For monocular sequences, we usually use relatively short hidden states because of limited memory resources. Therefore, the impact of the hidden state becomes very important. We follow Maslov & Makarov (2020a) and Patil et al. (2020) and use two training stages. On the first stage, we train a model for hidden states considering them as parameters, at the second stage we use learned hidden states as initial states for every sequence.

All models are trained for 20 epochs with batch size 2, where one sample in the batch is one sequence of length 10. We use the Adam optimizer with β₁ = 0.9 and β₂ = 0.999, learning rate 10⁻⁴ for the first 15 epochs which is then reduced to 10⁻⁵ for the remainder. The smoothness term λ is set to 0.001, and SSIM weight α is set to 0.85. Following Godard et al. (2019), we use pretrained ImageNet weights for the encoder. We also apply the following augmentations: horizontal flips with 50% chance; brightness, contrast, saturation, and hue jitter with ranges of ±0.2, ±0.2, ±0.2, and ±0.1 respectively.

For the baseline model, we follow Godard et al. (2019): we take the U-net model, where the encoder is a ResNet18 (He et al., 2015), and the decoder consists of upconvolutional filters, use the average reprojection term for the loss function, and do not apply scaling to decoder outputs before computing the loss.

System configuration

All experiments were conducted using a free distribution of Anaconda with Python 3.8. The models were implemented with the Pytorch library (Paszke et al., 2019). We used an RTX6000 GPU with 32 GB RAM. The operating system was Ubuntu 18.04.5 LTS.

Ablation Study

In order to estimate the effect of each component that we use in our pipeline, we provide an ablation study of the proposed modifications. An ablation study implies changing various components of our model, and its results help to better understand how each component contributes to the overall performance in monocular training. We first compare models with single-image evaluation settings (Table 1). Then we evaluate our models on sequences. The results are illustrated in Table 2. We can see that the baseline, without any of our modifications and contributions from Godard et al. (2019), performs the worst.

There are five different modifications; all models have the ConvGRU block but they differ in the number of channels in decoder outputs, the use of the upscaling block, and the use of the Fusion block. In particular, models #1 and #4 have only ConvGRU layers, other models use the Fusion block and upscaling, and differ only in Fusion block type.

Since our model was trained on sequences, we hypothesize that if we make use of sequence information at test time, we can achieve higher depth prediction accuracy. As a result, we propose two evaluation settings: single-image evaluation and sequential evaluation. Single-image evaluation means that we process each test frame separately. The model, initialized with trained hidden states, takes a single test frame as input and produces a depth map. On the other hand, sequential evaluation implies that we run the model on the previous n frames before producing the depth map for the target test image. We experimented with different values of n and empirically found that 10 frames of history appear to be optimal. During sequential evaluation we “gather” temporal information from previous frames via hidden states and utilize this information in the final prediction for the test image. This sequential evaluation approach does reflect real-life scenarios well since in these scenarios the depth estimation component usually does have access to the frame history. As for the KITTI dataset, previous frames are available for all test images except for those at the beginning of a scene; for these frames, we use the maximum available number of previous frames during evaluation.

Despite the fact that most of our models still lose in the final quality in terms of some evaluation metrics to Monodepth2, we can make some interesting conclusions regarding the proposed components. For example, all models perform better when they use sequences during evaluation. The upscaling technique also gives a boost to the depth prediction quality. As for the Fusion block, residual self-attention units slightly improve the quality compared to residual convolutional units, as we expected. In contrast, it is hard to say that the use of multi-channel decoder outputs leads to significantly better results. Therefore, this component may be considered redundant, i.e., not contributing to the overall model performance.

In order to choose the best model, we focus on the RMSE metric. Since the errors are squared before averaging, the RMSE score gives a relatively high weight to large errors. Therefore, we consider RMSE to be more reasonable in the depth estimation task, where large errors are particularly undesirable. According to these considerations, we choose model #2 to be the best in our experiments.

Finally, we also provide results of speed tests on RTX6000 GPU. We find that our model runs in real-time with 124 ± 16 FPS on resolution 192 × 640 compared to DORN with 15 ± 3 FPS.

Results on the KITTI dataset

We compare our best model with previous single-image approaches such as Godard et al. (2019), Guizilini et al. (2020a); Casser et al. (2018) and multi-frame sequential approaches such as Kuznietsov, Proesmans & Van Gool (2021), McCraith et al. (2020) and Watson et al. (2021). The results are presented in Tables 3 and 4, respectively. We provide scores on the KITTI Eigen split with input resolution 192 × 640. In addition, we compare to state-of-the-art supervised models such as Fu et al. (2018) and Maslov & Makarov (2020a). We observe that our approach outperforms most single-image self-supervised models in terms of the RMSE score. Our model achieves worse metrics than some sequential approaches, but next, we show that the proposed method better in qualitative comparison.