Multimodal image fusion for enhanced vehicle identification in intelligent transport

Single source vehicle detection algorithms

Wang & Gu (2020) proposed a novel deep learning model to detect vehicles from aerial images. This approach used a region-based Convolutional Neural Network that is augmented with a feature pyramid network (FPN) to address the multiple scale vehicle detection challenge. The FPN enabled the model to extract features at different scale which proved useful in detecting small size vehicle within an urban environment. For real-time detection of vehicles from UAV imagery, Song & Chen (2022) proposed a lightweight network architecture introduced a lightweight network architecture for real-time vehicle detection in UAV imagery. Their model named Fast Vehicle used depth wise separable convolutions and an efficient feature fusion module to decrease the computational cost yet increase the accuracy. In their work Deng et al. (2018) address the problem of identifying small vehicles from aerial images through their proposed Multi-scale Object Proposal Network (MS-OPN). In order to capture various sizes of vehicles, they employ a hierarchical feature extraction mechanism for detecting proposals at different scales. The MS-OPN is integrated with fully convolutional network for the final detection and classification. Hanyu et al. (2024) proposed AerialFormer, a hybrid model integrating CNN and Transformer architectures for aerial image segmentation. The model employs a CNN Stem to retain high-resolution low-level features, a Transformer encoder for multi-scale feature extraction, and a multi-dilated CNN decoder for aggregating local and global contextual information. Evaluated on iSAID, LoveDA, and Potsdam datasets, AerialFormer achieved superior performance compared to state-of-the-art methods. In their model Tong et al. (2020) suggested adding attention mechanism to enhance the detection of vehicles under different aerial scenes. The proposed attention guided network employed spatial as well as channel-wise attention modules that help learn the attention on frames’ features while discarding the background data. The model reveals the improvement of detection rates in more complex environment with background and occlusion. Tahir et al. (2024) dealt with the issue of vehicle detection under different lighting conditions and weather conditions in UAV images. A novel domain adaptation approach is developed that is based on adversarial training in an attempt to compensate for differences in images and lighting conditions. Guo et al. (2024) proposed a context-aware vehicle detection framework for aerial imagery. The model includes a spatial context module, which helps capturing long-range dependencies at the different regions of the image. This made it possible to optimize the use of the contextual information that exists in form of roads structures and buildings layouts among others to enhance the detection. Wang & Gao (2025) introduced Scale-Frequency Detection Transformer for Drone-View Object Detection (SF-DETR), an end-to-end transformer-based framework tailored for drone-view object detection. The architecture incorporates a lightweight ScaleFormerNet backbone with Dual Scale Vision Transformer modules, a Bilateral Interactive Feature Enhancement Module, and a Multi-Scale Frequency-Fused Feature Enhancement Network to address challenges such as tiny objects, complex urban backgrounds, and scale variations.

Multi modal fusion for vehicle detection

Multimodal fusion technologies have been developed as viable approaches to improve target detectability when used in UAVs. Since multi-sensor integration is used in such techniques, it is possible to overcome the shortcoming of single-modality methods and achieve the highest detection performance in specific conditions. A new multi-modal fusion architecture for detecting vehicles in aerial images based on integrating RGB images with thermal infrared (IR) was introduced by Wang et al. (2022a). The study applied two-stream Convolutional Neural Network structure by modeling two data streams at different channels RGB and IR. The features from both modalities were integrated together with an attention model that allowed the modification of the contribution of either modality depending on the input information. Liu, Zhao & Lam (2021) compared the LiDAR point cloud data with RGB images for vehicle detection under complex scenes in urban environments. They applied a voxelization method that alters the format of the collected 3D point cloud and makes it compatible with the convolution procedure more typical for 2D data. These larger LiDAR features were then fused with the RGB features through the cross modality attention fusion where the model could utilize both LiDAR and RGB data. Xue et al. (2021) pointed out the difficulties of merging hyperspectral imagery with RGB data in order to enhance the recognition of the target. Their fusion framework used a spectral-spatial attention mechanism for learning relevant features from hyperspectral data cube. These features were then combined with RGB features using gated fusion module that enabled the network to decide the contribution of each modality. The authors proved that the proposed approach greatly improved the identification of concealed objects and the ability to distinguish between similar objects in the visual field. Florea & Nedevschi (2025) introduced TanDepth, a practical scale recovery framework for converting relative monocular depth estimates into metric depth in UAV applications. The method projects sparse Global Digital Elevation Model measurements into the camera view using intrinsic and extrinsic parameters, then employs an adapted Cloth Simulation Filter to isolate ground points from the estimated depth map for scale alignment. This model-agnostic approach enables accurate metric depth recovery at inference time, regardless of the depth estimation network used. Shao et al. (2025) proposed UA-Fusion, an uncertainty-aware multimodal data fusion framework for 3D multi-object detection in dynamic traffic environments. The method introduces a probabilistic cross-modal attention mechanism within an uncertainty-aware fusion decoder to explicitly model and exploit aleatoric and epistemic uncertainties. It further incorporates an uncertainty-reduced object query initialization strategy, leveraging 2D detections to improve small-object recognition, and a query denoising-based robustness optimization strategy to enhance training stability under uncertain conditions. Dudczyk, Czyba & Skrzypczyk (2022) proposed a multi-modal fusion approach using visible light cameras and event-based sensors for high-speed target detection in UAV imagery. Event-based sensors have high temporal resolution and low latency that makes them suitable to work in parallel with spatial information provided by conventional cameras. A hybrid neural network is used that captures asynchronous event data along with frame based images for the detection purpose even in adverse environmental conditions. Rahman et al. (2024) presented a comprehensive review of UAV datasets, highlighting their diversity in modalities, including satellite imagery, drone-captured images, and videos. The study categorized datasets into unimodal and multimodal types, outlining their critical role in applications such as disaster damage assessment, aerial surveillance, object recognition, and tracking. Wang et al. (2022b) developed a multi-modal fusion approach which integrates RGB imagery and radio frequency (RF) signals to improve the target detection and classification in UAV context. Their framework used a new attention based fusion strategy for fusing features which were extracted form the visual data and those obtained from the RF spectrograms. This approach allowed for identification and categorization of targets that are in line of sight and targets that cannot be seen by the UAV, thus enhancing capability of the UAV based surveillance systems.

Image preprocessing

During the pre-processing of aerial image we had applied bilateral filter and gamma correction so as to enhance the quality of the image to be analyzed further. Bilateral filtering is non-linear image filtering technique that reconstructs the smooth image interacting the edges, based on pixel intensity and spatial location. This filtering process helps to reduce noise and enhance important structures within the image (Ravikumar et al., 2021). It can be represented by following eq.

(1) $$I\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{W}\left(\mathrm{x}\right)}\sum _{\mathrm{y}\in \mathrm{\Omega }}\mathrm{I}\left(\mathrm{y}\right).{\mathrm{f}}_{\mathrm{r}}\left(\left|\mathrm{I}\left(\mathrm{x}\right)-\mathrm{I}\left(\mathrm{y}\right)\right|\right).{\mathrm{f}}_{\mathrm{s}}\left(\left|\mathrm{x}-\mathrm{y}\right|\right)}$$here I(x) represents the intensity at pixel value x, W(x) is the normalization factor, and $f_r$ and $f_s$ represents the range and spatial Gaussian functions, respectively. After that we applied gamma correction in order to adjust the luminance of the images. It enhances the contrast of the regions with varying brightness, by applying a non-linear transformation. The gamma correction is defined as:

(2) $$I_{out}\left(x,y\right)={\left({\displaystyle \frac{I_{in}\left(x,y\right)-I_{min}}{I_{max}-I_{min}}}\right)}^{\gamma }\cdot (I_{max}-I_{min})+I_{min}$$here $I_{out}={\mathrm{I}}_{\mathrm{i}\mathrm{n}}$ is the output intensity, $I_{in}$ represents the input intensity, and γ is the gamma value which is used to control the level of correction. ${\mathit{I}}_{\mathit{m}\mathit{a}\mathit{x}}$ represents the maximum possible intensity value. A value of $\gamma <1$ increases brightness and on ther other hand $\gamma >1$ darkens the image. These preprocessing steps improve the image quality which helps in better feature extraction and segmentation in the later stages of our analysis (Guo & Wei, 2023).

Depth images generation

We propose a novel approach to generating depth maps from aerial images using a deep learning-based architecture. Our model is designed to estimate the depth of each pixel in an aerial image by utilizing a combination of convolutional layers and attention mechanisms. We integrate a self-attention module and Vision Transformer (ViT) components into the depth estimation model to capture both local and global dependencies in the image. Our custom depth estimation model, uses an encoder-decoder architecture enhanced by a self-attention mechanism. The encoder has two convolutional layers utilizing Rectified Linear Unit (ReLU) activation and max-pooling layers, succeeded by a self-attention mechanism that use the Multi Head Attention module including four attention heads. The feature extraction process is enhanced by this attention methods as it focus on critical regions of the image. In the decoder, we use two up sampling layers in order to increase the spatial resolution. The output of these layers are concatenated with corresponding encoder features (Zhou et al., 2021; Liang et al., 2022). The depth estimation process based on our model can be mathematically described as

(3) $$\hat{D}\left(x,y\right)=f_{dec}\left(concat\left(f_{enc}\left(I\left(x,y\right);W_e\right),A\left(f_{enc}\left(I\left(x,y\right);W_e\right),A\right)\right);W_d\right)$$here $I\left(x,y\right)$ represents the input aerial image, $\hat{D}\left(x,y\right)$ denotes the predicated depth maps, $f_{enc}(I\left(x,y\right);W_e$ represents the extracted feature map with the help of convolution layers, the self attention applied on the encoded feature is represented by $A$, $f_{dec}$ is the decoder that reconstructs the depth map from the concatenated encoder and attention features. The parameters $W_e,Aand{W}_d$ represent the learnable weights of the encoder, attention, and decoder, respectively. Additional convolutional layers then refine the upsampled features to ensure a smooth transition to the output. The last layer is another convolutional layer with sigmoid activation which produces a depth map ranging between 0 and 1. Self-attention is an important part of the model, and mathematically it can be described as:

(4) $$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V.$$

Here Q, K, and V are the query, key, and value matrices, respectively, and $d_k$ is the dimension of the key vectors. This mechanism allows the model to focus on relationships between different spatial regions of the feature maps. The model processes input images of size 224 × 224 × 3 through the encoder-decoder architecture which produce depth maps with the same spatial dimensions as the input. Post-processing is then applied to the generated depth maps in order to enhance their quality. On depth maps, bilateral filtering is used to preserve edges while smoothing depth values (Yang et al., 2021). This process can be described as:

(5) $$D_{filtered}\left(x,y\right)={\displaystyle \frac{\sum G_s\left(\left|\left|p-q\right|\right|\right)G_r(|I_p-I_q|)D\left(q\right)}{W_p}}$$where, Gs and Gr are the spatial and range kernel functions, Ip and Iq represent the intensities of pixels p and q, and Wp is the normalization factor. This filtering step ensures that fine details are maintained in the depth map. The model is displayed in Fig. 2.

Few of the depth generated images fro RGB are shown in Fig. 3 which helps in differentiating object surfaces and depths. The top row displays RGB images of various intersections and roundabouts, while the bottom row presents their corresponding depth images. In the RGB images, shadows and lighting variations can be observed, especially in areas where objects, such as trees and vehicles, cast shadows on the ground. These shadows are absent in the depth images, allowing a clearer view of structural details without interference from lighting. The depth images reveal spatial structure by emphasizing object boundaries, which are particularly useful for distinguishing objects from their background in complex urban scenes.

Image fusion: RGB and depth

Multi-modal fusion of RGB and depth images makes target detection in aerial imagery more precise, as the rich visual information of the RGB stream is complemented with spatial information from the depth stream. This approach leverages the complementary strengths of both modalities. RGB images contain color and texture information for distinguishing objects, while depth maps contain the specific structure of a scene for better spatial awareness. Thus, the obtained scene representation with richer contextual information enhances target detection and localization especially with respect to the vehicle type in the complex scene including occlusion and scale changes. The fusion process is carried out via guided image filtering and integrates RGB and depth data in a manner that can be controlled based on the image content. This edge-aware fusion helps in making sure that target boundaries do not get blurred in the fused image, which again is important for any detection and localization process (Sun et al., 2021; Wang et al., 2022b). The fusion process involves two basic step the first step is about the initial alignment in which RGB and depth images are first spatially aligned to ensure pixel wise correspondence which is done by using below given equation.

(6) $$F_{align}\left(x,y\right)=T\left(RGB\left(x,y\right),D\left(x,y\right)\right)$$here, $T$ represents the transformation function which aligns the RGB images with depth images at each pixel position $\left(x,y\right).$ The second step is responsible for the actual fusion process in which RGB image serves as the guidance image for filtering the depth images and can be represented by the equation.

(7) $$q_i=a_k\cdot I_i+b_k,\mathrm{\forall }i\in w_k$$where $q_i$ is the output pixel, $I_i$ is the input pixel, and $a_k$ and $b_k$ are locally adaptive linear coefficients. These coefficients are computed based on local image characteristics, such as edges, to ensure that the fusion happens in a way that preserves important structural information. The window $w_k$ represents the local neighborhood around each pixel where the guided filtering takes place. Moreover the ${\mathit{a}}_{\mathit{k}}$ and $b_k$ coefficients can be calculated using.

(8) $$a_k={\displaystyle \frac{1}{\left|\omega \right|}\sum {\displaystyle \frac{(I_i\cdot D_i)-{\mu }_k\cdot {\mu }_d}{{\sigma }_k^2+ϵ}}}$$

(9) $$b_k={\mu }_d-a_k\cdot {\mu }_k.$$

Here, $D_i$ represents the depth images while ${\mu }_k,{\mu }_d$ is the mean of the RGB images in local window $w_k$ and mean of depth images respectively. ${\mathit{\sigma }}_{\mathit{k}}^2$ is the variance, $\mathrm{ϵ}$ represents the regulization parameter and $\left|\mathit{\omega }\right|$ represents the size of local window. Finally the final fused image is represent by

(10) $$F=\alpha \cdot RGB+\left(1-\alpha \right)\cdot q.$$

Here $\alpha$ is the weighing parameter and it is used to control the contribution of each modality. This fused representation is beneficial in performing a better segmentation for target detection. The additional utilization of depth cues as part of the multi-modal features yields better discriminative power for the segmentation algorithms, which then enables clear differentiation of the targets from similar-looking non-target objects. For example, height profile of a vehicle relative to the surrounding can enhance the segmentation accuracy by features. The multi-modal fusion aids in handling challenging scenarios such as partial occlusions, varying lighting conditions, and diverse target orientations (Ahmed & Jalal, 2024b). The depth information is more robust to illumination variation and the fusion of the RGB and the depth data enables better reasoning about targets that may only be partially occluded. Figure 4 provides a comparative view of RGB, depth, and fused images for various urban scenes, including roundabouts and intersections. Each row represents a specific scene, with the first column showing RGB images, the second column showing depth images, and the third column showing the fused results. In the RGB images (first column), certain details are obscured by lighting variations and shadows such as trees and vehicles blending into shaded areas making it challenging to distinguish some objects. The depth images (second column), however, capture structural information independent of lighting, presenting a clearer view of object boundaries. For instance, in the depth images, such as the shape of the trees, vehicles, or other objects along the roadsides is clear even when the covering shadows hinder the visibility in the RGB images. The fused images (third column) are the best images which combine the variations of colors and depth from the RGB images and the structural depth. Previously hazy areas such as the road surface that seems to be shaded and vehicles that are partially hidden in these shadows are more easily outlined. This fusion process improves the scene interpretation since depth information can be added to the RGB view; this makes one to perceive distant objects and objects that are half hidden since the other half is hidden by the objects in front of it which cannot be perceived through RGB or depth alone.

Vehicle detection

We have used YOLOv4, a cutting-edge single-shot detector which is well known for its proficiency in detection, and classification tasks. YOLOv4 has the ability to perform these task by minimal number of training parameters, significantly enhancing its practical utility. The CSPDarknet53 backbone, presented in YOLOv4, uses the Cross Stage Partial (CSP) concept for better feature extraction and fewer computational expenses. This module integrates Cross Stage Partial Bottleneck (C3) and Efficient Layer Aggregation Network (ELAN) to improve portability and gradient flow information. The backbone includes the SPP module which ensure the accuracy for objects at different scales with reduced processing overhead. For highly effective feature fusion across scales, YOLOv4 integrates the PAN-FPN scheme in the neck section. This neck module has a head part based on YOLOv4, where the confidence and regression boxes are used to increase the level of accuracy using anchor-based detection (Amrouche et al., 2022). Non-Maximum Suppression (NMS) technique is used in the post processing of the detections for efficient handling in YOLOv4. The architecture of the YOLOv4 makes it possible for real-time objects detection, which has a high level of accuracy making it applicable in different field including traffic monitoring, cars without drivers, and surveillance systems. Thus, its flexibility in terms of completing multistep tasks with fewer parameters is another benefit for today’s object-detection purposes. With the help of enhanced components including CSPDarknet53, SPP, and PAN-FPN, YOLOv4 can be used as an Object Detection Tool, which features high performance and precision of outcomes (Mahto et al., 2020). The bounding box for can be represented by the following expression.

(11) $$\omega =p_w{e}^{t_w},h=p_h{e}^{t_h}.$$

Here, ${\mathit{p}}_{\mathit{w}}$ and ${\mathit{p}}_{\mathit{h}}$ are the width and height of the anchor box. The terms ${\mathit{t}}_{\mathit{w}}$ and ${\mathit{t}}_{\mathit{h}}$ are the predicated width and height offsets, while e denotes the exponential function. This equation is used in YOLOv4 to scale the anchor box dimensions to fit the sized detected object. By scaling the anchor box dimensions $p_w$ and $p_h$ with the exponential of the offsets $t_w$ and $t_h$, YOLOv4 can generate bounding boxes that match the size of the detected objects in the image. This process enables the model to adapt various sizes since the dimensions of the anchor boxes are changed depending on the feature learned from the image. We transfer the bounding boxes generated by YOLOv4 on the segmented images directly to the original images by maintaining the coordinate mapping. This helped to eliminate disconnection between the regions detected by the algorithm and the objects in the original images thus maintaining the precision of the detections in both image sets. For the detection process despite advancements in single-shot detectors we used YOLOv4 because it offers a unique balance between accuracy and computational efficiency, making it an optimal choice for our intelligent transportation system. The architecture of YOLOv4 effectively supports real-time vehicle detection without compromising the level of precision needed for reliable results in aerial imagery, as demonstrated by its superior performance on key datasets, including Roundabout, AU-AIR, and Vehicle Aerial Imagery Dataset (VAID). Moreover, the Cross Stage Partial connections in the architecture of YOLOv4 helps it perform better in regards to changes in viewpoints, scale, and orientation which are quite distinctive in aerial images. The framework of YOLOv4 for object detection model is displayed in Fig. 5.

Figure 6 presents vehicle detection results achieved using the YOLOv4 model. YOLOv4 detects multiple vehicles in the image by using bounding boxes, applying a one-stage detection mechanism for real-time efficiency. The multiple bounding boxes represent the identified vehicles to highlight the model’s performance of perceiving different vehicles with different sizes in a range of aerial images.

Feature extracion

We propose a novel feature extraction approach that integrates Histogram of Oriented Gradients (HOG) (Hussein et al., 2022) and Binary Robust Invariant Scalable Key points (BRISK) (Leutenegger, Chli & Siegwart, 2011) within the patch embedding layer of the Vision Transformer (ViT). Combining deep leaning with the classical computer vision approach, this hybrid method improves the ViT’s capability to capture not only the global context of an image but also the local structural cues (Naseer & Jalal, 2024). In our approach, each input image $X\in R^{H\times W\times C}$ where H, W, and C represent the height, width, and the number of channels (RGB), is first divided into non-overlapping patches of size P × P. The number of patches is:

(12) $$N={\displaystyle \frac{H}{P}\times {\displaystyle \frac{W}{P}.}}$$

Each patch $x_p\in {\mathbb{R}}^{P\times P\times C}$ is processed to extract deep and classical features within the Vision Transformer framework. Within the ViT patch embedding layer, for each image patch $x_p$, the following feature extraction steps are performed.

The HOG descriptor captures the gradient orientation distribution within the patch. For a patch $x_p$, the HOG features are calculated as:

(13) $$H_p={\displaystyle \frac{1}{N_H\cdot N_w}\sum _{i-1}^{N_H}\sum _{j=1}^{N_W}HOG\left(x_{p,i,j}\right).}$$

Here, $N_h\mathrm{a}\mathrm{n}\mathrm{d}{N}_w$ represented the number of horizontal and vertical block within each patch $x_p$. $HOG\left(x_{p,i,j}\right)$ represented the extracted features from (i,j) block of the patch ${\mathit{x}}_{\mathit{p}}$ and ${\displaystyle \frac{1}{{\mathit{N}}_{\mathit{H}}\cdot {\mathit{N}}_{\mathit{w}}}}$ is used to normlize feature from entire patch.

In parallel, the BRISK key points and descriptors are extracted from each patch $x_p$. The BRISK algorithm detects scale-invariant key points and generates corresponding descriptors

(14) $$B_p=\sum _{i=1}^{nk}{b}_i\cdot w\left(k_i\right).$$

Here $b_i$ is the brisk descriptor for keypoints $k_i$. $w\left(k_i\right)$ is a weight associated with keypoint $k_i$, based on its intensit and $nk$ resents the number of key point detected for each patch.

The extracted HOG and BRISK features are then embedded directly into the ViT’s patch embedding layer. Instead of passing only the flattened patch into the transformer, we concatenate the patch features with the HOG and BRISK descriptors to create an augmented patch embedding. The combined vector is projected into the transformer embedding space.

(15) $$z_p=W_e\left[x_p^{\mathrm{\prime }}\Vert H_p\Vert B_p\right]+b_e$$where $W_e{\mathbb{R}}^{\left(d+d_h+d_b\right)X{d}^{\mathrm{\prime }}}$ is a learnable embedding matrix $b_e$ is the bais term and $z_p\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$ represents the patch embedding passed to the transformer. After the patch embeddings are augmented with HOG and BRISK features, they are passed through the standard ViT architecture, where self-attention mechanisms operate on the combined feature vectors. This allows the transformer to compute attention across patches with enhanced local feature information from both HOG and BRISK descriptors. The final output of the transformer includes the Classification (CLS) token, which encodes the global representation of the image, taking into account both the global and local features.

(16) $$Z_{CLS}=W_{cls}\cdot Z_{input}+b_{cls}.$$

Here ${\mathit{Z}}_{\mathit{C}\mathit{L}\mathit{S}}$ is the output representation for the CLS token, which serves as the global summary of the entire input image, $W_{cls}$ is the weight matrix associated with the classification token, $Z_{input}$ is the input feature vector (the concatenated patch embeddings with HOG and BRISK features) and $b_{cls}$ is the bias term. This hybrid architecture effectively combines local texture information from HOG and BRISK with the ViT’s ability to model long-range dependencies, leading to improved performance in tasks. The extracted features are then normalized and plotted class wise in order to clearly visualize the feature distribution across different classes in datasets. The extracted features are visualized per dataset, showing distinct clusters that represent various vehicle classes.

ResNet-18 based classification

In our vehicle classification system, we utilized a Vision Transformer to extract discriminative features. These pre-extracted features are then fed into a modified ResNet-18 for precise classification. The ResNet-18, adapted for this task, leverages its deep architecture and unique residual blocks specifically for classification, rather than feature extraction. We made substantial alterations to the standard ResNet-18 architecture to effectively accommodate pre-extracted features. The initial 7 × 7 convolution and max pooling layers were replaced with a custom fully connected layer designed to adapt the input feature dimensions. We retained the four main residual block groups, modifying their input channels to match the dimensions of the pre-extracted features, while the architecture continues to include adaptive average pooling and a fully connected layer for classification. The custom input fully connected layer is trainable, enabling it to learn the mapping from pre-extracted features to internal representations. Since the input features are high-level abstractions, extracted by the Vision Transformer, the first two residual block groups, typically responsible for low-level feature extraction, are bypassed in this configuration. The last two residual block groups and the final fully connected layer remain trainable, allowing the network to learn task-specific high-level features and refine classification boundaries (Zhao et al., 2022). The input adaptation layer for our modified network can be represented by:

(17) $$Z_0=\sigma \left(W_0\cdot x+b_0\right)$$where $Z_0$ is the output of the initial fully connected layer, $\mathit{\sigma }$ (ReLu) is the activation function, $W_0$ is the weight matrix, $x$ is the input feature vector, and $b_0$ is the bias term. The final classification layer remains the same. In the $l^{th}$ residual block, the output is given by:

(18) $$Y_l=Z_{l-1}+F_l\left(Z_{l-1},\left\{W_l\right\}\right).$$

With the residual mapping defined as:

(19) $$F_l\left(Z_{l-1},\left\{W_l\right\}\right)=W_{1,2}\cdot \sigma \left(BN\left(W_{l,1}\cdot \sigma \left(BN\left(Z_{l-1}\right)\right)\right)\right).$$

Here, $Z_{l-1}$ represents the input if the ith block, $W_{l,1}$ is the weight matrix of the convolution layer while $BN(\cdot )$ is the batch normalization. The final classification layers computes class probabilities using

(20) $$a=softmax\left(\left(W_f\cdot F_L\left(F_{L-1}\left(\dots F_2\left(F_1\left(\sigma \left(W_0\cdot x+b_0\right)\right)\right)\dots \right)\right)+b_f\right)\right).$$

Here, ${\mathit{F}}_{\mathit{L}}$ represents the residual mapping output of the last residual block which refines the input features from VT. ${\mathit{W}}_{\mathit{f}}$ is the weight matrix of the final layer, the term ${\mathit{b}}_{\mathit{f}}$ represents the biasness in the final layer and $\mathit{a}$ is the output probability distribution over the classes.

Hyperparameters for model components

Table 1 below presents the principal hyperparameters of each model component in the proposed framework for vehicle detection and classification in aerial images. These parameters were determined through a combination of literature-guided manual tuning and a limited systematic search. Initial values were drawn from prior work employing YOLOv4, Vision Transformer (ViT), and ResNet-18 for related detection and classification tasks. For sensitive parameters such as learning rate and batch size, a small-scale grid search was performed on the validation split to identify settings that maximized the F1-score while keeping inference speed within real-time processing targets. Other parameters, including patch size, number of transformer heads, and dropout rate, were retained from well-established configurations in the literature due to their proven generalization performance. This balanced approach ensured experimental rigor while respecting the computational constraints of our cloud-based GPU environment. The YOLOv4 module was applied for vehicle detection, the Vision Transformer (ViT) for feature extraction, and ResNet-18 for multi-vehicle classification. Hyperparameter values for each component including learning rate, batch size, confidence thresholds, and structural parameters are provided to reflect the configurations used in achieving high-accuracy results on the Roundabout Aerial, AU-Air, and VAID datasets.

Discussion

As indicated in Fig. 7, in precision, we outperform all benchmarks and obtain precise high values for each of the proposed approach on all three datasets (our Roundabout: 0.984; our AU-AIR: 0.962; our VAID: 0.974). In order to increase the reliability of the results, all of the presented detection performance metrics were calculated with the 95% confidence interval (CI). This study depicted a high level of statistical significance in all the three sets of data collected. Performances with Roundabout set were as follows where precision was at 0.984 (95% CI [0.975–0.990]), recall at 0.981 (95% CI [0.972–0.987]) and F1-score at 0.983 (95% CI [0.974–0.989]). In the same way, when using the AU-AIR dataset, we identified the precision of the model with being equal to 0.962 (95% CI [0.952–0.970]), while the recall rate was equal to 0.967 (95% CI [0.958–0.974]) and the F1-score being equal to 0.965 (95% CI [0.956–0.972]). Since the VAID dataset comprises of highly similar images, high precision of 0.974 (95% CI [0.965–0.981]) and recall of 0.972 (95% CI [0.963–0.979]) can be explained. It is clear from narrow confidence intervals across all metrics and on all three datasets the statistical reliability and reproducibility of our results. This validates the effectiveness of our proposed approach. These enhancements can be directly addressed in practical area of Intelligent Transportation System where traffic flow can be controlled in real time manner, incident can be identified and traffic pattern can be analyzed.

Practical impact

This work introduces a unique aerial vehicle detection and classification framework that integrates attention-based monocular depth generation, edge-aware guided RGB–depth fusion, YOLOv4 detection, and a hybrid Vision Transformer enhanced with HOG and BRISK descriptors. While each component has been explored individually in prior studies, their combination in this configuration is, to our knowledge, novel for aerial vehicle recognition. The synergy between depth-aware fusion and hybrid global local feature encoding improves both detection and classification accuracy, particularly for small and occluded vehicles. Beyond its technical contributions, the proposed system offers clear practical value for traffic surveillance, road safety monitoring, and incident detection, delivering high performance and efficiency suitable for real-time cloud-based or edge-computing deployment.

Failure cases

In Fig. 8, we present several failure cases where the model was unable to detect vehicles. In part (a), the red arrow highlights a vehicle missed due to truncation, where only a partial portion of the vehicle is visible within the frame. In part (b), the model fails to detect a car that is heavily occluded by a tree, making its features less distinguishable. Upon closer inspection, such errors tend to occur more frequently for smaller vehicle classes (e.g., motorcycles and bicycles), at higher flight altitudes where object resolution decreases, and in scenes with dense vegetation or strong shadows. Certain camera angles, particularly oblique perspectives, also increase the likelihood of misdetections due to shape distortion.

To mitigate these issues in future work, we plan to (1) incorporate multi-scale feature enhancement modules specifically tuned for small-object representation, (2) explore synthetic data augmentation simulating occlusion, truncation, and varied lighting conditions, (3) employ multi-view or temporal data to recover partially visible vehicles, and (4) integrate adaptive attention mechanisms capable of focusing on low-contrast and partially visible targets. Expanding the dataset to cover diverse geographical regions, road types, and environmental scenarios will further improve robustness. Additionally, domain adaptation strategies such as adversarial learning can be applied to fine-tune the model for varying landscapes without extensive labeled data. These improvements will help enhance detection consistency and reliability across different operational conditions, supporting broader real-world deployment in intelligent transportation and smart city systems.