Applying auxiliary supervised depth-assisted transformer and cross modal attention fusion in monocular 3D object detection

Image-only monocular 3D object detection

Most previous monocular detectors follow the principles of conventional 2D detectors (Ren et al., 2016; Detector, 2022; Lu et al., 2021). Because single images inherently miss depth information, geometric consistency plays a major role in determining the location and orientation of objects. This reliance involves utilizing the known geometric properties of objects and their spatial relationships within the scene to reflect three-dimensional features from 2D information. However, the absence of direct depth cues often limits the accuracy and reliability of these predictions, making it challenging to achieve precise 3D object localization and orientation estimation.

As early works, Deep3Dbox (Mousavian et al., 2017) addresses orientation prediction with a novel multi-bin loss, which discretizes orientation angles for better accuracy. By using geometric priors, Deep3Dbox also enforces constraints between bounding boxes from different dimensions, ensuring consistency between the 2D projections and 3D predictions. Monocular 3D Region Proposal Network (M3D-RPN) (Brazil & Liu, 2019) predicts 3D object bounding boxes using 2D constraints and incorporates convolutional layers to improve the accuracy of 3D object predictions. This approach combines spatial localization guidance with depth-related feature extraction to enhance overall detection performance. Orthographic feature transform is proposed in Optical Fiber Temporal Network (OFTNet) (Roddick, Kendall & Cipolla, 2019). OFTNet projects image-based features into a 3D voxel space using orthographic projections, thereby improving the spatial representation of objects. By directly transforming features into a volumetric representation, OFTNet improves the model’s ability to capture three-dimensional spatial information from monocular images. MonoPair (Chen et al., 2020a) focuses on improving detection performance by exploring spatial pairwise relationships between objects. MonoPair analyzes the orientations of targets and their positions within the scene, enhancing the context-awareness of the detection process. This spatial analysis aids in more accurately localizing and classifying objects, especially in complex visual environments. Additionally, key points are utilized in many works like Li et al. (2020), Liu, Wu & Tóth (2020) and Luo et al. (2021), as an indicator locating the targets. However, these image-only monocular methods often face challenges in accurately localizing objects due to their exclusive dependence on 2D images, which do not provide direct depth information. This limitation can lead to challenges in precisely determining object positions and dimensions in 3D space, especially in complex scenes with varying depths and occlusions. As a comparison, our Depth-DETR adopts LiDAR input as extra supervision. With the help of accurate depth information from LiDAR features, Depth-DETR has overcome the shortage of image features only providing 2D information.

Depth-assisted monocular 3D object detection

The inherent challenge posed by lacking of depth in images has spurred the development of depth-assisted models aimed at improving detection performance. These models leverage additional information from depth estimation networks or employ geometric reasoning techniques to enhance the accuracy of object localization and classification.

Depth-guided Dynamic-Depthwise-Dilated Local Convolutional Network (D4LCN) (Ding et al., 2020) and Depth-Conditioned Dynamic Message Propagation for Monocular 3D (DDMP-3D) (Yin, Zhou & Krahenbuhl, 2021) pay attention to approaches based on fusion. These approaches integrate images with estimated depth information using convolutional networks. These methods aim to leverage the complementary strengths of 2D image features and depth cues to enhance 3D object detection performance. By combining these data sources, D4LCN and DDMP-3D upgrade the accuracy and robustness of object localization and classification in three-dimensional space. Although both models incorporate additional depth estimation modules to assist in acquiring more geometric information, the ultimate supervised target remains the detection box. The predicted results from the depth estimation module cannot be effectively validated. Inaccurate depth estimation results can indeed impact the overall effectiveness of the model. To address this issue approaches such as Categorical Depth Distribution Network (CaDDN) (Reading et al., 2021) and Monocular 3D Object Detection with Depth-Aware Transformer (MonoDTR) (Huang et al., 2022) employ additional supervised targets. These methods integrate supplementary supervised learning objectives to increase the accuracy and reliability of monocular object detection. CaDDN focuses on learning categorical depth distributions to generate bird’s-eye-view features. By leveraging these features, CaDDN recovers bounding boxes from the projected view, improving the accuracy of object localization and orientation estimation in monocular settings. MonoDTR generates ground truth depth supervision by leveraging LiDAR data. This approach enhances monocular 3D object detection by utilizing precise depth information obtained from point cloud data, which is then used to supervise and increase the accuracy of depth estimation in the detection process. Recently, Depth Equivariant Network for Monocular 3D Object Detection (DEVIANT) (Kumar et al., 2022), A Unified Vehicle and Infrastructure-side Monocular 3D Object (MonoUNI) (Jinrang, Li & Shi, 2024) and some other works (Qin & Li, 2022) explored the approach of depth translations in the projective manifold, showing greater abilities of cross-datasets. SeaBird (Kumar et al., 2024) emphasized the ability of detecting large objects, effectively integrating BEV segmentation on foreground objects for 3D detection, with the segmentation head trained with the dice loss. However, projecting LiDAR and other 3D data to pixel-level depth supervisions is relying on pre-trained projecting models. These pre-trained models cannot guarantee absolute accuracy, and incorrect supervision targets will similarly lead to deviations in depth estimation from real results, thereby affecting the final accuracy of the model. Depth-DETR utilized depth information in the other way. Extra supervisions of depth estimator could provide more accurate depth feature, then enrich the final features. While other methods of utilize depth estimator could provide inaccurate depth informations.

Fusion methods in multi-modal 3D object detection

With the development of different sensors, 3D object detection is moving forward to multimodality. Comparing to single-modal models, multimodal models are able to utilize images, depth information, point clouds, and other type of data together for object detection. However, this approach faces challenges of integrating multimodal data. Instead of tacking or adding features together, an increasing number of researches is focusing on how to effectively and efficiently fuse multimodal data using attentions.

Single-stream multimodal Transformer like Universal Image-text Representation Learning (UNITER) (Chen et al., 2020b) and Vision-and-Language Transformer (ViLT) (Kim, Son & Kim, 2021) typically consists of a series of multimodal self-attention modules at each layer. These models facilitate early and extensive interaction and fusion of different modalities. Tokens representing different modalities are combined through concatenation. Each layer’s attention mechanisms can focus on both intra-modal (within-modal) and inter-modal (across-modal) contexts. This approach enables the model to effectively utilize and integrate information from various modalities throughout the network’s depth, enhancing its ability to capture comprehensive contextual dependencies both within and across modalities. However, flattening and concatenating multimodal features can result in excessively long feature vectors, which negatively impacts computational efficiency. In order to overcome the disadvantages of self-attention, MonoDETR (Zhang et al., 2023) and Bird’s-eye View representations with spatiotemporal transformers (BEVFormer) (Li et al., 2022) employ cross-attention mechanisms to combine multi-modalities features. In the cross-attention mechanism, queries from one modality are used to attend to keys and values from another modality, facilitating the exchange of information between the two modalities. Following this initial fusion, self-attention layers are applied to the merged features for further refinement, allowing the model to deeply integrate and contextualize the combined information. Despite the advantages of cross-attention in merging heterogeneous features, this method has some limitations. The fusion directions in cross-attention are predetermined and fixed, which can restrict the interaction strength between distant modalities. This rigidity can limit the model’s ability to dynamically adapt to varying degrees of relevance and dependency between the features of different modalities. Depth-DETR introduces a parallel cross modal attention fusion mechanism. With this design, the features extracted by Depth-DETR not only complement the two different depth features but also enhance the interaction and fusion between visual and depth features, enriching the final features with 3D spatial information.

Feature extraction

The image $I\in {\mathbb{R}}^{H\times W\times 3}$ is inputted into the network with its height and width. Depth-DETR adoptes feature backbone to generate four levels of feature maps in size $\frac{1}{4}$, $\frac{1}{8}$, $\frac{1}{16}$, and $\frac{1}{32}$ of original input image size.

Depth-assisted transformer encoder

The depth-assisted encoder of Depth-DETR is composed of a depth encoder supervised by ground-truth depth, a depth encoder supervised by bounding boxes, and a visual encoder.

Decoder of Depth-DETR

As three features input to the decoder of Depth-DETR, we first combine depth feature $f_{Dgt}^e$ and $f_{Dbbox}^e$ with $f_V^e$ respectively. In order to fuse depth features and visual features, which are different modalities with different feature sizes. We first introduce cross modal attention.

Cross modal attention

To combine two modalities which are visual and depth, the features are defined as $X_{visual}\in {\mathbb{R}}^{t_{\alpha }\times d_{\alpha }}$ and $X_{depth}\in {\mathbb{R}}^{t_{\beta }\times d_{\beta }}$, where $t_{\alpha }=H_{\alpha }\times W_{\alpha }$ and $t_{\beta }=H_{\beta }\times W_{\beta }$.

$W_{Q_{visual}}$, $W_{K_{depth}}$ and $W_{V_{depth}}$ are defined as the learnable transformation matrix in attention mechanism. Therefore, the fusion process from depth to visual feature is shown as:

(3) $$X_{fused}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{X_{visual}{W}_{Q_{visual}}\times W_{K_{depth}}^{\mathrm{\top }}{X}_{depth}^{\mathrm{\top }}}{\sqrt{d_k}}\right)X_{depth}{W}_{V_{depth}}.$$

We call (Eq. (3)) a single-head cross modal attention, which is illustrated in Fig. 3.

Detection heads and loss

After the decoder, the queries are processed through heads combined by MLP layers to predict various 3D attributes, including object category, 2D size, projected 3D center, depth, 3D size, and orientation.

The losses of these six attributes are categorized by two groups. As these attributes primarily pertain to the 2D visual appearance of the image, object category, 2D size, and the projected 3D center are included in the first group $\mathcal{L}2D$. As a contrary, 3D spatial properties of the object like depth, 3D size, and orientation are set as the other group $\mathcal{L}3D$.

In detection transformer approaches, the number of queries is set as N, and $N_{gt}$ represents the number of ground-truth objects. We firstly match $N_{gt}$ valid pairs out of N, and the overall loss is formulated as:

(5) $${\mathcal{L}}_{overall}=\frac{1}{N_{gt}}\cdot \sum _{n=1}^{N_{gt}}({\mathcal{L}}_{2D}+{\mathcal{L}}_{3D})+{\mathcal{L}}_{dmap},$$where ${\mathcal{L}}_{dmap}$ represents the Focal loss (Lin et al., 2017) of the predicted depth map $D^{predicted}$.

Settings

Comparison

Visualization

As depicted in Fig. 4, examples from the KITTI test dataset illustrate our findings. In the figure, the green bounding boxes denote the detected objects, while the red dashed rectangles highlight false and missed detections, and the red solid rectangles indicate correct detections. Compared to our baseline on the top, Depth-DETR leverages depth encoders and cross-modal attention to enhance key feature information in local details of challenging objects during detection as a comparison in the bottom. This underscores the effectiveness of incorporating supervised depth information and a refined fusion mechanism, which improves the model’s capability to understand three-dimensional spatial information within images.

Ablation study

As the core depth encoder, we explore how to better depth encoder to interact with final feature in Table 4, by only utilized one or two of the encoders. Because of the deeper transformer network, the visual feature is the feature contains more information among three features, while depth encoders are limited by the lightweight design of network. Therefore, if the network only takes one feature to perform 3D object detection, the visual feature will receive the best performance. The $A{P}_{3D}$ of visual feature is 18.95, improved by +2.67% and +4.19% comparing to other two depth features.

Although both depth encoder work well with the visual features, depth feature supervised by ground-truth depth performs better due to its extra supervision. As shown in the result, two types of depth feature have different draws of their own. One due to the inaccurate point cloud projections, the other due to the unequal supervise targets. However, combining two depth features together can balances each others limitations and generate more effective performance. In the result, combining three features together could give an increase up to +7.56%.

In Table 5, we experiment three different fusion methods. Self-attention, as a method for extracting information from a single feature, requires flattening and concatenating the three different modality features into a new feature. When visual features are the primary focus and the two depth features are secondary, the concatenated feature becomes excessively long and contains too much irrelevant information. Because of this drawback, using self-attention to fuse features only results in 28.64 in $A{P}_{3D}$. Stacking cross-attention and self-attention together could solve this problem, but manually setting the sequence of fusion does make it hard to find attention connections among modalities, especially modalities which do not perform cross-attention directly. Therefore, stacked cross attention was receives improvement of +0.38%. However, since cross-modal fusion attention can only fuse data from two different modalities at a time, sequential fusion is necessary when dealing with three modalities. The fixed order of sequential fusion results in lower interaction intensity between distant modalities, hindering effective integration of data from all three modalities. As a comparison, using cross modal attention to parallel fuse two depth features with visual features helps to avoid early interactions. This method with 30.32 in $A{P}_{3D}$, proving that combining two fused features with attention mechanism and using self-attention to perform the final fusion do balance two types of depth features.

In Table 6, we explore different depth positional encoding schemes for $f_{Dgt}^e$ and $f_{Dbbox}^e$ within the cross modal attention. From the beginning, we test without any embedding. It is interested to find out that traditional sinusoidal functions for 2D encoding shows a lower precision in 3D object detection, due to lacking abilities representing 3D information.

As shown in the results, the meter-wise encodings represented as $p_D$ approach outperforms other encoding methods by effectively capturing fine-grained depth cues across the range from $d_{min}$ to $d_{max}$, reaching improvements of +3.56% and +3.84% compared to the other two methods. This method enriches the queries with comprehensive scene-level spatial structures, thereby enhancing performance.

To prove the reproducibility of Depth-DETR, we now list out the five runs of MonoDETR and our Depth-DETR in Table 7, it shows that Depth-DETR outperformsMonoDETR in all runs and in the average run.