DeepFusion encoder for unsupervised monocular metric depth estimation

Self-supervised depth estimation

Traditional depth estimation methods depend heavily on a substantial amount of labeled data for supervised learning (Kuznietsov, Stuckler & Leibe, 2017; Chen et al., 2016; Eigen & Fergus, 2015; Cao, Wu & Shen, 2017). However, acquiring depth labels in real-world scenarios is both challenging and costly, which has led to the rise of self-supervised learning approaches. In recent years, unsupervised deep learning has made remarkable advancements in monocular depth estimation, particularly when accurate depth labels are limited. Self-supervised methods present an effective solution for depth estimation tasks (Garg et al., 2016; Godard, Mac Aodha & Brostow, 2017). In the realm of self-supervised depth estimation, the most commonly used re-projection error is defined as $L_{photo}=\sum _s\sum _t\Vert I_s-I_t{\Vert }_1$, where the image synthesized from the depth map and camera pose is denoted as $I_s$, while $I_t$ represents the target image. The re-projection error is optimized by minimizing the pixel differences between these images to refine depth estimation.

One of the commonly used methods in self-supervised depth estimation was based on stereo vision techniques, which achieved depth estimation through disparity map prediction (Li et al., 2024; Cheng, He & Zhang, 2018; Mo et al., 2023; Chen et al., 2021; Xia et al., 2024). In this approach, the model learns the disparity from the left image to the right image and uses geometric constraints to derive the depth map. $\mathrm{D}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}=b\cdot f\mid \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mid$, where $b$ is the baseline between the left and right cameras and $f$ is the focal length of the camera. The advantage of this method lies in its independence from explicit depth labels; relatively accurate depth maps can be generated through simple geometric transformations. However, this approach is contingent upon the use of a stereo camera system, where the horizontal distance b between the two monocular cameras must be meticulously calibrated; any inaccuracies in this measurement can adversely affect the precision of depth estimation. In practice, such precision may be significantly compromised by factors such as vibration and deformation, resulting in substantial deviations. To mitigate these discrepancies, recalibration is essential. Although online calibration methods are currently available, their accuracy remains uncertain, which imposes certain constraints on practical applications.

With the advent of monocular depth estimation methods, researchers have increasingly explored self-supervised learning approaches within monocular video sequences by leveraging temporal contextual information. A prominent example of these approaches is based on view synthesis. These methods predict the relative pose between the current frame and its adjacent frames, utilizing the depth map to synthesize images from alternative viewpoints. The model’s training objective is to minimize the re-projection error between the synthesized image and the actual image, thereby eliminating the necessity for direct use of depth labels during training. The article “Unsupervised Learning of Depth and Ego-Motion from Video” (Zhou et al., 2017) was pivotal in opening this field, leading to several subsequent enhancements aimed at improving performance. Notable advancements include the introduction of more sophisticated geometric models and a 3D convolutional depth estimation framework designed to enhance depth prediction accuracy (Casser et al., 2019; Zhang, Zhang & Tao, 2022; Bian et al., 2019). With the advancement of deep neural networks, self-supervised depth estimation methods have undergone significant evolution. Researchers are increasingly exploring ways to incorporate additional sources of information and semantic data within the self-supervised framework to enhance both the accuracy and robustness of depth estimation (Zhang, Zhang & Tao, 2022; Chen et al., 2019; Clark et al., 2017). These advancements suggest that self-supervised depth estimation can achieve performance levels comparable to those of supervised learning, all without relying on costly labeled datasets.

Metric monocular depth estimation

Depth estimation from monocular camera poses can be accomplished through various methodologies. Triangulation involves matching feature points across images and utilizing camera poses to compute three-dimensional (3D) points. Structure-from-motion (SfM) reconstructs 3D scenes by detecting features, estimating camera poses, and triangulating points across multiple frames (Lou & Noble, 2024; Guo et al., 2025). Optical flow estimates depth by analyzing pixel motion between frames in conjunction with camera movement, while disparity calculates depth based on pixel differences, focal length, and baseline distance (Teed & Deng, 2020; Shao et al., 2022). The selection of a particular method is contingent upon the specific application and the data available. However, in practice, these methods are often influenced by scene complexity and noise levels, which can result in diminished estimation accuracy. With the advancement of deep learning, metric depth estimation methods that leverage deep neural networks have gradually emerged. These approaches typically involve the integration of additional sensor data or specific training strategies to learn the absolute depth scale (Park, Kim & Sohn, 2019; He et al., 2024; Wang et al., 2024). For instance, certain studies combine LiDAR data with visual information, utilizing the depth measurements provided by LiDAR to calibrate the scale of depth estimation and thereby achieve precise metric depth estimation. However, this methodology also presents several drawbacks, including the high costs associated with LiDAR sensors, increased system complexity, and potential performance degradation (e.g., calibration accuracy), which collectively limit its widespread applicability.

In the field of depth estimation, metric monocular depth estimation seeks to directly derive depth maps with an absolute scale from a single image (Piccinelli et al., 2024; Zhang, Zhang & Tao, 2022; Wofk et al., 2023). Unlike traditional relative depth estimation methods, this approach requires the capability to determine actual distances between objects and the camera by integrating sensor information. This ability is essential for various practical applications, including autonomous driving and robotic navigation. However, due to vehicle vibrations and complex external environments, scaling models often encounter difficulties in accurately adapting to these conditions. As a result, discrepancies in computed outcomes emerge, posing challenges in ensuring precision in depth estimation.

Self-supervised absolute depth estimation methods have not been extensively researched or widely implemented, primarily due to challenges associated with guaranteeing their accuracy and generalization capabilities. These limitations hinder their effectiveness in real-world scenarios. The strength of this approach lies in its capacity to utilize a vast number of unlabeled images for learning purposes, thereby eliminating the need for costly annotated data and reducing data acquisition expenses. Nevertheless, accuracy may be compromised by variations in scene dynamics and lighting conditions that can negatively impact the reliability of depth estimation.

To address these challenges, our approach incorporates kinematic geometric constraints between image frames, along with integrated metric scale information that includes motion cues such as vehicle speed, acceleration, and angular velocity. This enhancement significantly improves both the accuracy and robustness of absolute depth estimation.

Depth estimation of dynamic objects

Self-supervised monocular depth estimation relies on the geometric consistency between frames, where static scenes enable more accurate pixel matching. However, the presence of dynamic objects compromises this assumption by introducing displacements that are independent of self-vehicle motion. As a result, the model encounters difficulties in establishing reliable correspondences for moving objects, often leading to erroneously inflated depth predictions that diverge toward infinity.

To tackle this challenge, various methodologies have been investigated. One such approach is the automask technique, which analyzes pixel information between frames to identify moving objects and subsequently excludes them from depth estimation (Zhou et al., 2017). While this method enhances accuracy for static objects, it constrains the ability to estimate depth for dynamic entities. Deep learning approaches utilize convolutional neural networks (CNNs) to identify dynamic objects and estimate depth based on the global context of a scene (Cheng, He & Zhang, 2018). While these methods are capable of learning intricate depth relationships from large datasets, they often demand substantial computational resources and extensive labeled data. Graphical model approaches, as exemplified by Vogel, Schindler & Roth (2013), capture the motion trajectories of dynamic objects to facilitate depth estimation. Although effective in managing motion relationships, these models can become complex and computationally intensive, particularly in highly dynamic environments. Recent advancements in neural radiance fields (NeRFs) have expanded their capabilities to encompass dynamic scenes, thereby enabling the depth estimation of moving objects. NeRF-based methods, such as those proposed by Li et al. (2021) and Park et al. (2021), employ implicit neural representations to model both static and dynamic components within a scene. By learning a continuous volumetric representation of the environment, these methods can estimate the depth of dynamic objects without necessitating explicit motion segmentation. However, NeRF-based approaches frequently encounter challenges related to real-time performance due to their computationally demanding rendering processes. Furthermore, they require dense multiview input and may struggle to generalize effectively to unseen motion patterns or highly occluded scenes (Tretschk et al., 2021).

In response to these limitations, this study proposes a novel approach that integrates optical flow geometric methods with multi-scale adaptive neural networks. The depth decoder is designed to progressively transform high-level feature representations generated by the encoder into dense depth maps through multiple upsampling stages. By effectively incorporating multi-scale contextual information from the encoder, the decoder can merge fine-grained details with a comprehensive understanding of the global scene, resulting in depth predictions that are both accurate and robust. The depth decoder employs skip connections from corresponding layers of the encoder. These connections ensure that high-resolution details from earlier layers are preserved and integrated into the final depth map. This method significantly enhances the accuracy of depth estimation and effectively addresses challenges associated with estimating depth for dynamic entities.

Figure 2 illustrates our framework pipeline. It presents consecutive raw image frames acquired from the camera. A deep fusion network is employed as our backbone, enhancing both efficiency and accuracy. The optical flow loss is utilized for estimating depth in dynamic scenes. We incorporate reprojection loss and speed loss to facilitate metric depth estimation. The final output includes the depth estimation result, represented as a disparity map, along with point cloud data derived from this disparity map.

The proposed approach utilizes hierarchical context encoding (HCE), which extracts features at multiple scales through the application of multi-scale convolutions. The feature attention module is employed on the combined feature maps generated by the hierarchical context encoding module. To further enhance representation, a global feature aggregation layer captures the overall scene context and conveys this information to the decoder. This facilitates dynamic, geometry-aware decoding that sharpens geometric details and improves boundary accuracy, ultimately resulting in the generation of four multi-scale inverse depth maps. In parallel, we integrate an optical flow decoder to predict dense motion fields, thereby introducing dynamic constraints that further refine depth estimation precision. By amalgamating multiscale context encoding, comprehensive scene understanding, and motion-guided constraints, our architecture achieves robust and highly accurate depth estimation.

Preliminaries

In this section, we provide an overview of reprojection loss, optical flow, and the commonly employed loss functions within the context of monocular depth estimation.

Innovative convolution neural network

Previous research has primarily focused on improving the accuracy of depth estimation by emphasizing the volume and quality of supervised data, as well as the complexity of neural network architectures. This study aims to develop a network capable of effectively capturing both global context and fine-grained details within images while ensuring real-time inference. As illustrated in Fig. 3, the proposed framework consists of an encoder-decoder architecture referred to as DepthNet, along with a PoseNet module. The DepthNet is responsible for generating inverse depth maps at multiple scales from input images, whereas the PoseNet predicts the relative camera pose between consecutive frames. Subsequently, the model reconstructs the target image and computes a loss function to guide learning, ultimately facilitating the estimation of absolute-scale depth.

The proposed DeepFusion Network primarily comprises DepthNet and PoseNet, along with a comprehensive loss function. DepthNet is organized into four stages that include both encoding and decoding processes. Before the concatenation of each layer, a max pooling operation is performed. Following downsampling, the HCE module and attention block operations are applied prior to employing skip connections. This sequence is succeeded by convolutional layers, upsampling, and additional convolutions to generate depth estimation features. PoseNet shares the Resnet50 architecture with DepthNet; although it may demonstrate slightly slower training speeds, it excels in capturing intricate image detail features. The loss function integrates reprojection loss, optical flow loss, and speed loss.

The proposed network architecture incorporates Hierarchical Context Encoding through initial layers consisting of Conv2D, BatchNorm, ReLU, and MaxPool. This is followed by residual blocks that facilitate the extraction of multiscale features. The encoded features undergo further processing via four context processing modules with varying dilation rates, specifically designed to enhance multiscale contextual representation. This integrated design ensures robust depth estimation while improving precision and enabling fine-grained detail recovery.

The operations ‘conv1’, ‘bn1’, ‘relu’ and ‘maxpool’ collectively form the initial convolution block of the network, responsible for extracting low-level features from the input images, such as edges and textures (Liu, Wan & Gao, 2024; Panda et al., 2022; Ming et al., 2021; Targ, Almeida & Lyman, 2016). Given an input image $\mathbf{I}\in {\mathbb{R}}^{H\times W\times C}$, the initial convolution block applies a transformation $f:{\mathbb{R}}^{H\times W\times C}\to {\mathbb{R}}^{H^{\mathrm{\prime }}\times W^{\mathrm{\prime }}\times D}$ where:

(10) $$f(I)=\mathrm{Re}LU(BN(\mathrm{Conv}(I))).$$

Here, $W^{\mathrm{\prime }}$ and $H^{\mathrm{\prime }}$ denote the reduced spatial dimensions after downsampling and are the depth of the output feature. This transformation aids in preserving critical feature details by minimizing spatial redundancy, thereby facilitating efficient feature extraction for subsequent layers. These layers progressively increase the number of channels in the feature maps (width) while simultaneously reducing the spatial dimensions, ultimately forming high-level semantic features. In layer 4, a multigrid mechanism and dilated convolutions are employed to expand the receptive field, enabling the network to capture broader contextual information from a larger area.

Hierarchical context encoding module

Unlike traditional methods that utilize context encoding in image segmentation, we incorporate multiscale HCE modules into the depth estimation pipeline. This integration enables the network to more effectively address depth discontinuities and large-scale variations in depth (Zhang et al., 2018; Nguyen, Fookes & Sridharan, 2020; Chen et al., 2018). The HCE module extracts hierarchical features through multiscale convolutions. By expanding the receptive field, it captures contextual information across multiple scales, thereby facilitating accurate depth estimation for objects situated at varying distances. In conventional convolutional operations, the growth of the receptive field is gradual as the kernel size increases. However, with hierarchical context encoding, we can enhance the receptive field without augmenting the number of parameters. The convolution operation is defined as follows:

(11) $$y[i]=\sum _{k}x[i+r\cdot k]\cdot w[k].$$Regarding Eq. (11), $x$ is the input feature map, $w$ is the convolution filter, $r$ is the multi-scale factor, $y$ is the output feature map, $i$ is the spatial index in the input feature map. When set the different multi-scale factor $r$, the convolution kernel operates at multiple spatial scales, allowing it to aggregate information from a broader context while preserving both the number of parameters and feature map resolution. This is crucial for capturing context at different scales without pooling layers, which typically reduces resolution.

HCE employs multiscale feature extraction with a multi-scale factor D set to 1, 3, 6, and 12. A global feature aggregation layer integrates global scene context and subsequently feeds this information into the decoder for dynamic geometry-aware decoding. This process refines geometric details and enhances boundary precision, resulting in the generation of four inverse depth maps at multiple scales. Additionally, we utilize an optical flow decoder to estimate dense motion fields, incorporating dynamic constraints that significantly improve depth estimation accuracy. This architecture effectively combines multiscale context, global scene understanding, and motion-guided constraints to achieve precise and robust depth estimation.

The module applies several parallel convolutions with different multi-scale factor, and then concatenates the results. The formula for HCE’s output is as follows:

(12) $$y_{HCE}=\mathrm{Concat}\left(y_1,y_6,y_{12},y_{18},y_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}\right).$$

Regarding Eq. (12),

• $y_1$ is the result of a $1\times 1$ convolution (captures local information).

• $y_6$, $y_{12}$, $y_{18}$ are the results of $3\times 3$ convolutions with multi-scale factor of 6, 12, and 18, respectively.

• $y_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}$ is the output of the global average pooling, which captures the global context by aggregating the features throughout the image.

Figure 4 illustrates the hierarchical context encoding and attention mechanism. Each branch of the HCE module is designed to capture features at varying scales, thereby enhancing the network’s ability to effectively manage objects of different sizes in a single pass. This capability addresses a significant challenge in monocular depth estimation.

Multi-scale feature fusion and enhancement

In the context of monocular depth estimation, extracting features at multiple scales is crucial to capture both fine details and global structures of the scene. However, simply extracting features at different scales may not be sufficient for complex scenarios. Thus, it is important to enhance and fuse these multiscale features to ensure that the model can effectively utilize the information at different levels. To enhance the features extracted by the encoder, hierarchical context encoding modules are employed in the feature extraction process (Jie, Yian & Bin, 2024). The modules are designed to capture context at multiple scales by employing convolutional operations with varying receptive field sizes, allowing the network to focus on both local details and broader contextual information. This ensures that the model can better handle varying depth ranges within a single scene.

Figure 5 presents a comparison of different processing stages through three images. The first image represents the original input, devoid of any processing; the second image overlays the feature map extracted using the standard ResNet feature extraction algorithm, which illustrates the fundamental structural features of the image. The third image incorporates a feature map enhanced by the hierarchical context encoding algorithm, emphasizing intricate details and complex patterns within the image. This demonstrates the advantages of this approach in enhancing feature representation.

Speed fusion for absolute depth estimation

In monocular depth estimation, a significant challenge is the scale ambiguity problem. This issue arises because the network can estimate relative depth but lacks the capability to predict metric depth (Wang et al., 2024; Zhang, Song & Lou, 2024; Bian et al., 2019; Liu et al., 2015; Johnston & Carneiro, 2020; Gordon et al., 2019). We addressed this challenge through speed supervision, leveraging speed measurements obtained from the inertial measurement unit (IMU) to inform the predicted depth. However, since speed derived from the IMU results from integrating acceleration data, it tends to drift over time, leading to inaccuracies in both speed and angle estimations. Our proposed speed supervision technique integrates IMU data with information regarding the vehicle’s own speed. This approach not only ensures that the predicted depth maps maintain metric consistency across different sequences but also enhances the accuracy of absolute depth estimation.

Given a vehicle’s ground truth speed $v_{\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}}$, the network estimates the depth by predicting the relative motion between frames. The speed can be inferred from the predicted depth and pose as

(21) $$v_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}}=\frac{\mathrm{\Delta }d}{\mathrm{\Delta }t}$$where:

• $\mathrm{\Delta }d$ is the distance traveled by the vehicle, calculated from the translation vector ${\mathbf{T}}_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}}$ (the pose predicted by the network).

• $\mathrm{\Delta }t$ is the time interval between frames (typically known from the timestamps).

(22) $$\mathrm{\Delta }d=\Vert T_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}}[:3,3]\Vert$$Here, $T_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}}[:3,3]$ represents the translation vector between the two frames.

UMAD dataset

In order to enhance the validation of our algorithm, we not only utilized the KITTI dataset but also constructed a UMAD dataset aimed at improving the accuracy and generalization capabilities of our model. The hardware components employed in this study include FMCW LiDAR, a monocular camera, and an IMU sensor.

The advantages of our LiDAR system compared to the KITTI LiDAR are threefold: (1) The 120-line LiDAR offers greater line density and higher accuracy than the KITTI’s 64-line configuration, achieving a horizontal resolution of 0.1 degrees and a vertical resolution of 0.08 degrees; (2) FMCW LiDAR is capable of obtaining velocity values for each point on dynamic objects, thereby providing valuable insights into how these dynamic entities may influence estimation results in monocular depth estimation—this feature distinguishes it from other datasets such as KITTI, CityScapes, and nuScenes; (3) It can measure both the relative velocity of dynamic objects with respect to the LiDAR itself as well as their absolute velocity relative to the ground.

Monocular camera parameters: The camera is a high-definition monocular camera with a frame rate of 25 Hz and an image length and width of 640 $\times$ 480. Intrinsic parameter of the camera, $f_x$ = 703.64, $f_y$ = 724.82, $c_x$ = 322.55, $c_y$ = 117.60. Extrinsic parameter from LiDAR to camera, $x$ = 0.0, $y$ = 0.0, $z$ = 0.080000. To reduce calibration errors, the camera is installed on top of the LiDAR and concentric with the LiDAR

IMU parameters: The system employs a high-precision 6-degree-of-freedom IMU sensor, which facilitates real-time and accurate acquisition of vehicle pose. This sensor is installed on the upper part of the LiDAR and is positioned 50 mm along the X-axis from the center point of the LiDAR. The IMU sensor plays a crucial role in LiDAR calibration and data synchronization. While it can also provide integral speed information, its accuracy is inferior to that obtained from the vehicle’s CAN port.

The UMAD dataset distinguishes itself from other datasets such as KITTI, CityScapes (Cordts et al., 2016), and nuScenes (Caesar et al., 2020) by utilizing FMCW LiDAR technology (Kim, Jung & Lee, 2020; Wu et al., 2024; Gu et al., 2022). This advanced technology not only captures point cloud data but also delivers velocity information for each individual point. The dataset is available in ROS bag file format and includes a dedicated topic for dynamic points, rendering it highly valuable for research in autonomous driving, robotics, and computer vision. As illustrated in Fig. 6, this integrated sensor system is mounted on the vehicle.

The UMAD dataset comprises approximately 200 GB of data, organized into five distinct scenarios: urban roads, enclosed parks, underground garages, commercial district roads, and night scenes, as illustrated in Fig. 7. Each scenario has a duration of about 10 min. In the figure, the rows from top to bottom correspond to urban roads, enclosed parks, and underground garages respectively. The columns from left to right display the image data alongside the corresponding 3D point cloud data.

Datasets

The models were developed and trained using a robust dataset comprising 79,620 images, which were subsequently evaluated on an additional set of 8,848 images. This extensive dataset was meticulously curated to encompass diverse scenes and lighting conditions, thereby ensuring that the model learns from a wide range of depth estimation scenarios. The training and evaluation images are high resolution, measuring 640 $\times$ 192 pixels, and were captured using a monocular camera system with intrinsic parameters normalized to the image size: specifically ( $f_x$ = 0.58), ( $f_y$ = 1.92), and principal point offsets ( $c_x$ = 0.5), ( $c_y$ = 0.5). The preparation of the dataset involved generating ground truth depth maps utilizing LiDAR data from the KITTI dataset, employing sensors to ensure accurate depth supervision. The models underwent fine-tuning on the KITTI raw dataset, where both left- and right-camera images were utilized to reflect the stereo setup. For ground truth depth map generation, LiDAR data were processed with a specific calibration matrix and resized to maintain consistency in training and evaluation. In addition to conducting training evaluations on the KITTI dataset, we also employed the UMAD dataset—which includes FMCW LiDAR data alongside inputs from a monocular camera and an IMU sensor—to enhance both model accuracy and ablation studies.

Implementation details

The hardware configuration utilized in our experiments comprises an Intel® Xeon(R) CPU E5-2680 v4 operating at 2.40 GHz across 28 cores, accompanied by 64 GB of RAM, a 2TB SSD, and an RTX 4060 Ti GPU with 16 GB of memory. The depth estimation component of our project is implemented using the PyTorch framework. Depth maps are computed on the GPU under the following experimental parameters: input images are resized to a height of 192 pixels and a width of 640 pixels. The network architecture is based on the DeepFusion Network and has been trained on the KITTI and UMAD dataset. The training process employs a batch size of 12 and a learning rate set to 1e−4, conducted over a total of 20 epochs. A disparity smoothness weight of 1e−3 was applied, with depth predictions normalized within a range from 0.1 to 90 m. The model initialization incorporates pre-trained weights, while pose estimation is executed through a separate decoder. Training encompasses various loss functions including re-projection loss, smoothness loss, optical flow loss, affine invariant loss, and velocity loss derived from IMU and vehicle fusion data. Our framework also facilitates multiscale training with scales configured as [0, 1, 2, 3]. These parameters were meticulously selected to optimize performance for the specified hardware setup; all computations are efficiently executed on the GPU. Notably, the entire training process requires approximately 10 $\sim$12 h when conducted on a single NVIDIA RTX 4060 Ti GPU.

We evaluated the performance of our self-supervised monocular depth estimation approach through both quantitative and qualitative comparisons with state-of-the-art methods across various scenes (Liu, Salzmann & He, 2014; Eigen, Puhrsch & Fergus, 2014). For the quantitative assessment, we employed multiple metrics that are commonly utilized in prior research. Performance of the model using standard metrics Abs Rel, Sqr Rel, RMSE, RMSE (log) and $\delta <1.25$, $\delta <{1.25}^2$ and $\delta <{1.25}^3$, respectively, as follows:

• Abs Rel: The mean absolute relative error, calculated as $\frac{1}{N}{\sum }_i\frac{|d_i-p_i|}{d_i}.$

• Sqr Rel: The mean squared relative error, defined as $\frac{1}{N}{\sum }_i\frac{{(d_i-p_i)}^2}{d_i}.$

• RMSE: The linear root mean square error, $\sqrt{\frac{1}{N}{\sum }_i{(d_i-p_i)}^2}.$

• RMSE^(log): The logarithmic root mean square error, $\sqrt{\frac{1}{N}{\sum }_i{(\mathrm{l}\mathrm{o}\mathrm{g}{d}_i-\mathrm{l}\mathrm{o}\mathrm{g}{p}_i)}^2}.$

• Threshold: The percentage of pixels where $\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{d_i}{p_i},\frac{p_i}{d_i}\right)<\delta ,$ indicates the fraction within the defined error threshold.

Here, $p_i$ represents the predicted depth for each pixel $i$, and N denotes the total pixel count. We trained the monocular depth network using outdoor scenes, thereby aligning our methodology with recent state-of-the-art approaches. While indoor datasets were excluded from this study, it is important to note that similar training can also be conducted utilizing indoor data.

Depth estimation performance

To systematically evaluate the performance of deep estimation algorithms, we employ a combined approach that integrates qualitative visual comparisons with quantitative metric calculations to comprehensively validate the predictive accuracy of the models.

Figure 8 presents a qualitative comparison of monocular depth estimation performance, showcasing DeepFusion alongside previous methodologies using frames from the KITTI dataset (Eigen test split). Our approach exhibits an improved capacity to capture finer details and structures—such as those present in vehicles, pedestrians, and slender poles—attributable to its multi-scale learned preservation of spatial information. Our models were trained for 20 epochs at a resolution of 640 × 192 with a batch size of 12. In contrast, SfMLearner (Godard, Mac Aodha & Brostow, 2017) underwent training for 200 epochs at a resolution of 416 × 128 with a batch size of 4. For Lite-Mono (Zhang et al., 2023), training was performed over the course of 30 epochs at a resolution of 640 × 192 utilizing the same batch size of 12. Similarly, Monodepth2 (Godard et al., 2019) was trained for 20 epochs at a resolution of 640 × 192 while employing an identical batch size of 12.

The depth estimation process employs a multiscale decoder along with a dynamic scale factor to produce accurate and scale-consistent depth maps from monocular images. As illustrated in Fig. 9, (a) represents the original image, (b) depicts the disparity map, and (c) shows the disparity map converted into a point cloud. In contrast to previous self-supervised monocular depth estimation methods (Godard et al., 2019; Chen et al., 2016; Zhou et al., 2017; Zhang, Zhang & Tao, 2022), our model broadens the scope of depth estimation to encompass both outdoor and indoor environments. The images were captured in a spontaneous manner across diverse environments, including libraries, offices, factory interiors, and airport terminals.

As illustrated in Table 1, DeepFusion demonstrates significantly superior performance. The comparative data are sourced from Monodepth2 (Godard et al., 2019), R-MSFM (Zhou et al., 2021), Lite-Mono (Zhang et al., 2023), and Eite-Mono (Ren, 2024). Depth Error ( $\downarrow$) indicates that a lower value is preferable, while Depth Accuracy ( $\uparrow$) signifies that a higher value is desirable (Shi et al., 2024; Peluso et al., 2020; Ai & Wang, 2024). “M” denotes KITTI monocular videos; “M^∗” refers to an input resolution of 1,024 $\times$ 320; and “M $†$” indicates models trained without pre-training on ImageNet (Deng et al., 2009). The best results are highlighted in bold.

Metric depth estimation evaluation

In Fig. 10, the original image data and ground-truth data were sourced from the KITTI dataset, specifically identified as 2011_09_26_drive_0022_sync. The images are numbered sequentially from top to bottom as follows: 26_001_l, 34_215_l, 22_473_r, and 28_497_l. Each row is arranged from left to right, displaying the original image, the point-cloud data derived from the disparity map, and in the final column, the corresponding ground-truth point-cloud data. In this experiment, the speed_loss_weight was set to 0.2 with a total of 10 epochs and a learning rate of 1e−4. Three points were selected in each image; specific coordinate values can be found in Table 2. By comparing these coordinate values, we can assess whether there is consistency between the estimated depth values and their actual counterparts.

The data presented in Table 2 correspond to Fig. 10, arranged from top to bottom. Three points, denoted as p0, p1, and p2, were selected from the generated depth estimation point cloud map, and their coordinates were recorded. Similarly, three corresponding points—p0(GT), p1(GT), and p2(GT)—were extracted from approximately the same locations within the ground truth point cloud data. Although selection of point positions may not achieve complete precision, the deviation in depth values remains within a margin of 2.0 m. This level of accuracy is deemed sufficient for practical perception applications in specific scenarios. The results indicate that our DeepFusion network effectively mitigates the inherent scale ambiguity associated with monocular systems, ensuring that the generated depth maps are consistent with real-world scales.

To enhance our understanding of the dispersion in depth estimation, we compute the average Euclidean distance, standard deviation, and standard error of the target points.

In Fig. 11, randomly sample depth estimation images from the evaluation dataset and convert them into pseudo-point clouds. Randomly extract point coordinates from the point cloud and compare them with the ground truth point cloud data. Calculate the average Euclidean distance $\mu$, standard deviation $SD=\sqrt{\frac{{\sum }_{i=1}^n{\left({\mathbf{x}}_i-\mu \right)}^2}{n-1}},$ and standard error $SE=\frac{\mathrm{S}\mathrm{D}}{\sqrt{n}}.$ The unit of the vertical axis is meters.

Ablation studies

Table 3 employs checkmarks $✓$ to signify the adoption of a method, while crosses ✗ indicate that a method is not adopted. The term “training hours” refers to the total duration from the commencement to the conclusion of the training process. Our models were trained for 20 epochs at a resolution of 640 $\times$ 192 with a batch size of 12. The best results are emphasized in bold font.

Speed evaluation and GPU memory occupation

We conducted experiments to evaluate the memory and inference speed parameters of the proposed model utilizing the KITTI dataset. The model achieves an impressive inference speed of 80 frames per second on videos with a resolution of 640 $\times$ 192, when deployed on an RTX 4060Ti GPU. Utilizing the disparity maps generated by the model, we performed conversions from 2D to 3D point clouds as well as object perception experiments. As demonstrated in Table 4, the results indicate that the model meets the real-time object perception requirements essential for autonomous driving applications. Furthermore, the maximum recorded GPU memory usage is 1,192 MB, which renders it suitable for deployment on standard vehicle-mounted controllers.

Practical application

Figure 12 illustrates the application of monocular depth estimation, employing a model trained on the DeepFusion network. Initially, the model generates a disparity map, which is subsequently converted into a point cloud through camera interpolation, thereby facilitating three-dimensional target perception. The findings indicate that DeepFusion is capable of achieving precise metric depth estimation for 3D perception of both static and dynamic objects. This capability effectively addresses the requirements of practical perception applications across a diverse array of scenarios.