Enhanced fire detection: a weighted bounding box fusion approach based on Dempster–Shafer theory

Computer vision for fire detection

The rising urbanization has heightened the risk of fires, necessitating improved detection approachs. While traditional smoke sensors cover large areas, video/image surveillance systems offer a valuable solution for identifying smoke and flames from a distance. Recent advancements have led to the development of various techniques aimed at enhancing fire detection (Özel, Alam & Khan, 2024). As highlighted in Shamta & Demir (2024), the advancement of drone applications in firefighting has largely centered around the remote sensing of forest fires. Aerial monitoring of these fires is both costly and carries considerable risks, particularly in the context of uncontrolled wildfires. Islam & Hu (2024) propose an autonomous system for forest fire monitoring designed to quickly track specific hot spots. They develop an algorithm to direct drones toward these hot spots, validated through a realistic simulation of forest fire progression. Similarly, Beachly et al. (2018) discuss aerial tasks such as prescribed fire lighting. However, the practical implementation of these tasks is still limited by logistical challenges, particularly in transporting large quantities of water and fire retardant. Kasyap et al. (2022) developed a framework that integrates mixed learning techniques (including the YOLO deep network and Light Detection and Ranging (LiDAR) technology) to effectively control a Unmanned Aerial Vehicle (UAV)-based fire detection system. This system can fly over burned areas and provide precise and reliable information. Ren et al. (2024) also proposed a real-time forest fire monitoring system that employs UAVs and remote sensing technologies. The drone is equipped with sensors, a mini processor, and a camera, allowing it to process data and images on-board from various sensors.

According to Ma et al. (2018), using UAVs for fire monitoring and detection in quasi-operational field trials proves more effective and safer than risking human lives. Given the complex and unstructured nature of forest fires, UAVs are capable of collecting extensive datasets. To analyze this data effectively, several machine learning techniques have been proposed. One such approach involves integrating color, motion, and shape features with machine learning algorithms for fire detection (Harjoko et al., 2022; Sudhakar et al., 2020). These methods take into account fire characteristics beyond just color, such as irregular shapes and consistent movements in specific areas. Alternatively, deep learning methods have been suggested to classify features like fire, no-fire, smoke, flame, and more (Khan et al., 2022a; Zhou et al., 2024; Choutri et al., 2023). For example, Khan et al. (2022a) introduce a forest fire detection system that focuses on accurately classifying fire versus non-fire images. They created a diverse dataset called Deep-Fire and leveraged Visual Geometry Group (VGG)-19 transfer learning, which demonstrated improved prediction accuracy compared to various other machine learning method. Choutri et al. (2023) investigate the use of YOLO models (versions 4, 5, 8, and Neural Architecture Search (NAS)) to detect fire, no fire, and smoke. Additionally, the authors discuss the fire’s localization using a stereo vision method.

Bonding boxes for object detection

Object detection plays a pivotal role in computer vision systems, with applications spanning autonomous driving, medical imaging, retail, security, facial recognition, robotics, and beyond. Modern approaches predominantly rely on neural network-based models to accurately locate and classify objects within predefined categories. While real-time inference is often a priority, there are scenarios where the primary focus is on enhancing detection accuracy. In such cases, employing ensembles of models has proven to be effective in achieving superior results. Authors in Solovyev, Wang & Gabruseva (2021) introduce an innovative technique for merging predictions from multiple object detection models: the WBF method. The approach leverages the confidence scores associated with each proposed bounding box to generate averaged boxes, resulting in a more accurate and reliable detection outcome. Qian & Lin (2022) introduce an innovative algorithm utilizing weighted fusion techniques to detect forest fire sources across diverse scenarios. Yolov5 and EfficientDet, two separate weakly supervised models, are combined in the methodology. The method handles datasets efficiently since training and prediction activities are carried out in tandem. The combination of these models produces an optimum fusion frame by processing prediction results using the WBF technique. An object detection model that carries out optimized coefficient-weighted ensemble operations on the outputs of three different object detection models—all trained on the same dataset—is presented in Körez et al. (2020). The proposed model is engineered to generate a more accurate and reliable output by applying an optimized coefficient-weighted ensemble to the results of multiple object detection methods. Li et al. (2022) introduce an innovative adaptive label assignment strategy called Dual Weighting (DW) to enhance the accuracy of dense object detectors. Unlike traditional dense detectors that rely on fixed, coupled weighting, DW breaks this convention by dynamically assigning separate positive and negative weights to each anchor. This is achieved through the estimation of consistency and inconsistency metrics from multiple perspectives, allowing for more precise and effective training of dense object detection models.

Data fusion

Data fusion has made significant strides in advancing early wildfire detection methods. By integrating data from multiple sources, such as thermal sensors, UAV cameras, weather data, and satellite imagery, it becomes possible to enhance the accuracy and speed of wildfire detection and response efforts. This approach allows for a more comprehensive understanding of the fire’s behavior and location, enabling authorities to take proactive measures to mitigate its impact. In Zhao et al. (2014), proposed an adaptive weighted fusion algorithm based on Dempster–Shafer theory of evidence for early wildfires warning. Environmental data, including temperature, infrared temperature sensor data, and smoke data, they were subsequently homogenized and fused to derive a comprehensive assessment of fire incident. In Oliveira et al. (2022), a data fusion based decision support system for forest fire prevention and management has been proposed. This system incorporates a flexible Internet of Things (IoT) multisensor node equipped with temperature and humidity sensors, Global Navigation Satellite System (GNSS) receiver, Carbon Monoxide (CO) and Oxygen (O2) sensors and Infrared (IR) sensors for measuring and mapping the surrounding temperature. The collected data enable the identification of the hot spots and the generation of precise wildfire trajectory predictions. In their study, Benzekri et al. (2020) employed a sensor fusion approach where a wireless sensor network was used to measure various parameters, such as temperature and CO levels. Subsequently, deep learning algorithms were applied to process the acquired data and determine the presence of a forest fire. In Rizogiannis et al. (2017), a decision level fusion approach based on Dempster–Shafer theory has been introduced for early fire detection. Data from meteorological and atmospheric sensors were combined with data from UAV infrared cameras and some other ancillary sources in order to generate early warning alerts and notifications. In El Abbassi, Jilbab & Bourouho (2020), a reliable method for rapid wildfire detection and fire spread estimation has been proposed, the approach is based on the processing of multi-sensor data fusion and decision-making. Temperature, humidity, and smoke data are collected, processed and fused using K-Nearest Neighbors (KNN) classifier and the fuzzy inference. In Kim & Lee (2021), a Bayesian network-based information fusion, combined with deep neural network approach was proposed for robust fire detection. Various environmental data from different sensors were combined with visual data from video sequences using simple Bayesian network. In Soderlund & Kumar (2023), an approach employing data fusion for estimating the real-time spread of wildfires has been proposed. Dempster–Shafer theory was applied to combine data coming from static temperature sensors embedded on the surface, and data collected from mobile vision sensors housed on unmanned aerial vehicles UAVs.

Comparison with related methods

Several post-processing strategies have been proposed in the literature to address the issue of overlapping bounding boxes in object detection. Among them, Non-Maximum Suppression (NMS) (Ren et al., 2017; Redmon & Farhadi, 2018) remains the most widely adopted due to its simplicity and computational efficiency. NMS discards boxes with lower confidence scores when they overlap significantly with a higher-scoring box. Although effective in many scenarios, prior studies (Bodla et al., 2017; Alabi et al., 2019) have reported that NMS often suppresses true positives in dense scenes, such as wildfire smoke plumes or clustered objects, leading to reduced recall.

Weighted Boxes Fusion (WBF) (Solovyev, Wang & Gabruseva, 2021) has been introduced as an alternative that aims to overcome some of the limitations of NMS. Instead of discarding boxes, WBF merges them by computing a weighted average of their coordinates, where weights are proportional to the confidence scores. This approach can improve localization accuracy and preserve more potential detections, as demonstrated in applications like multi-model ensembling for object detection (Solovyev, Wang & Gabruseva, 2021; Bochkovskiy, Wang & Liao, 2020). Nevertheless, WBF employs a deterministic averaging process and does not explicitly account for prediction uncertainty, making it less robust when the input boxes contain contradictory or ambiguous information.

The proposed Dempster–Shafer Weighted Boxes Fusion (DS-WBF) builds upon the strengths of WBF while addressing its key limitation. By incorporating Dempster–Shafer theory (Shafer, 2020), the method provides a probabilistic reasoning framework that explicitly models uncertainty and manages conflicting evidence between bounding boxes. Unlike NMS and WBF, which primarily rely on geometric overlap and score heuristics, DS-WBF leverages belief functions to assign and combine degrees of support for different hypotheses (e.g., presence of fire), enabling more informed fusion decisions. This makes DS-WBF particularly advantageous in challenging conditions frequently reported in recent wildfire detection studies (Vasconcelos et al., 2024; Zheng et al., 2023), where visual evidence can be incomplete, noisy, or partially occluded.

In summary, while NMS is efficient and WBF improves localization through weighted averaging, the proposed DS-WBF uniquely combines geometric information, confidence weighting, and uncertainty reasoning, offering a more robust solution for post-processing in object detection pipelines.

Basic definitions

System development process

In this article, we proposed a fire detection system for earlier fire detection using computer vision. Figure 1 presents a comprehensive overview of the fire detection system development process using fire imagery, divided into two primary stages: data processing and fire detection. In the data processing stage, the process begins with the data collection step, where original fire images are gathered, serving as the foundational input for model training. Following this, the data preparation phase involves meticulous labeling of these images to annotate fire regions accurately. The data is then methodically split into training, validation, and testing sets, ensuring that the model training process is well-structured and reliable.

Moving to the fire detection stage, the process is further refined through model selection, training, and evaluation. In this phase, different models are selected, trained, and rigorously evaluated based on their performance in detecting fires. The detection approachs employed include Non-Maximum Suppression (NMS), Weighted Bounding Box Fusion (WBF), and an advanced approach known as Dempster–Shafer Weighted Bounding Box Fusion (DS-WBF). These approachs are crucial in refining detection outcomes, categorizing image regions as either Fire or No-Fire. This systematic approach highlights the importance of integrating advanced fusion techniques to enhance detection accuracy and reliability.

Data

Large and high-quality datasets are essential to the effectiveness of Yolo-based forest fire detection systems. A high-quality dataset enables deep learning models to capture a wide range of features, leading to better generalization capabilities. In our study, we curated an extensive dataset comprising over 12,000 images from various public fire datasets. This dataset features fire images captured under varying conditions and equipment setups. Representative examples from this dataset are presented in Fig. 2.

Before deploying the dataset, preprocessing steps were undertaken to ensure both the relevance and quality of the aerial images for our study. Initially, images that lacked visible fire or where the fire was indistinct were removed. The resulting dataset comprised 12,000 images drawn from state-of-the-art public fire datasets—BowFire (Barros, Silva & Santos, 2014), FiSmo (Dimitropoulos, Barmpoutis & Grammalidis, 2015), and Flame (Habiboglu, Gunay & Cetin, 2011)—as well as newly acquired images. These cover a diverse range of scenes, including urban areas, forested regions, indoor settings, and aerial perspectives, under varying lighting and weather conditions. Each image was meticulously labeled into two categories: Fire and No-fire. All images were then uniformly resized to $640\times 640$ pixels to standardize the input dimensions for the model. Manual labeling was performed to generate precise ground truth data, which includes both image filenames and their associated bounding box coordinates. This preprocessing pipeline was essential for ensuring dataset consistency and suitability for effective training.

The dataset needed to be divided into separate subsets in order to be ready for training machine learning models. To ensure a reliable assessment of the model’s performance, the dataset was split into an 80% training set and a 20% testing set. Data augmentation is another often used preprocessing approach that creates copies of the original dataset by applying affine changes, rotations, and scaling. By using this technique, the neural network is exposed to a wider variety of visual patterns and the dataset’s diversity is increased. In doing so, it improves the model’s capacity to identify objects in a variety of positions and forms, which in turn raises the accuracy of detection.

Yolo models

In the field of computer vision, You Only Look Once (YOLO) is a cutting-edge, real-time object recognition method that has received a lot of popularity. YOLO reframes object identification as a single regression issue, directly predicting bounding boxes and class probabilities from entire images in a single evaluation, in contrast to standard detection systems that apply a classifier to an image at several locations and scales. This approach not only improves detection speed but also enhances accuracy by leveraging global context (Jiang et al., 2022). For this article, three YOLO models: YOLOv3, YOLOv5, and YOLOv9 are used, each offer unique advantages for object detection tasks particularly in the context of fire detection.

YOLOv3, introduced in 2018, marked a significant improvement over its predecessors with a deeper architecture using Darknet-53 as its backbone. Its multi-scale detection approach makes it effective for identifying objects of different sizes, a critical feature in detecting fire and smoke. While YOLOv3 is known for its balance between speed and accuracy, it may lag behind more recent versions in terms of precision and efficiency, especially in complex environments (Jiao et al., 2019).

YOLOv5, although an unofficial continuation of the YOLO series, quickly gained popularity due to its modern design and user-friendly implementation. It introduced several key enhancements, including automated anchor box learning and better data augmentation techniques, making it faster and more accurate than YOLOv3. YOLOv5’s lightweight, modular design allows it to handle large-scale datasets and complex environments effectively, making it ideal for real-time fire detection. However, while it offers significant improvements, it may not fully utilize the latest deep learning advancements needed for the most complex scenarios (Jocher et al., 2021).

YOLOv9 represents the cutting-edge of the YOLO family, incorporating the latest techniques in feature extraction and model optimization. It aims to push the boundaries of detection accuracy, especially in challenging tasks where objects are small or overlapping. YOLOv9 excels in detecting and classifying objects in complex environments, making it well-suited for dynamic fire detection scenarios. However, its advanced capabilities require more computational resources, which might limit its accessibility for applications with limited hardware (Wang, Yeh & Liao, 2024a).

In comparing these models, the choice often depends on the specific application needs. YOLOv3 provides a solid foundation for general-purpose detection, but may struggle in more complex tasks requiring high speed and precision. YOLOv5 offers a balanced approach, making it attractive for real-time fire detection with its efficiency and ease of use. YOLOv9, while offering the highest performance, is best suited for environments with abundant computational resources and where cutting-edge accuracy is crucial. The different Yolo models architectures used for this article are desicpated in Table 1.

Training platform

For this study, we utilized Google Colab as our primary computational environment, chosen for its cost-effectiveness, accessibility, and the advanced hardware options it offers for machine learning tasks. Google Colab provides a default setup featuring an Intel Xeon CPU with 2 virtual CPUs (vCPUs) and 13 GB of RAM, which is sufficient for many deep learning experiments. Additionally, Colab offers access to a NVIDIA Tesla K80 GPU with 12 GB of VRAM, enabling efficient training of complex neural networks, particularly convolutional neural networks (CNNs).

The choice of Colab was informed by previous experiences with machine learning and artificial intelligence systems, where managing dependencies can often be challenging. Colab’s pre-configured environment alleviates much of this complexity, allowing researchers to focus on model development rather than environment setup. The ability to easily switch between CPU and GPU, reset the environment, and experiment with different configurations without the need for extensive package management makes Colab an ideal platform for rapid testing and prototyping in machine learning research. This flexibility is especially valuable in projects that require extensive experimentation and iterative development.

The training and validation curves in Fig. 3 confirm that the model was properly trained and converged in a stable manner. Both the training and validation box losses (train/box_loss and val/box_loss) decrease steadily across epochs, indicating continuous improvement in bounding box localization. The objectness losses (train/obj_loss and val/obj_loss) follow a similar downward trend, demonstrating the model’s increasing ability to distinguish fire-related regions from the background. Minor oscillations observed in the early epochs are typical in object detection training, arising from the adaptive balance between localization and classification objectives, and they gradually diminish toward convergence. The classification losses remain negligible throughout training, reflecting the well-defined categorical structure (fire, no-fire, smoke) and the adequacy of the dataset. Precision and recall curves exhibit an overall upward trend with higher stability in later epochs, while both Mean Average Precision (mAP)_0.5 and mAP_0.5:0.95 continue to improve, confirming strong generalization across IoU thresholds.

Importantly, the proposed Dempster–Shafer Weighted Boxes Fusion (DS-WBF) method remains effective even under slight training instabilities, as it explicitly manages uncertainty and conflicting evidence among detections. This robustness ensures consistent performance and reliable decision fusion, further validating the stability of the proposed approach.

Classification metrics

By averaging the precision-recall curve for every class, the Average Precision (AP) metric is utilized to assess how well a model performs in classification tasks. The confidence score indicates how likely it is that an object will be found in a particular anchor box. The Intersection over Union (IoU), a measure of overlap between predicted and ground truth bounding boxes, is computed. To discriminate between True Positives (TP) and False Positives (FP), an IoU threshold of 0.5 (50%) is used. One of the most important metrics for evaluating a model’s accuracy for a given class is precision, which is the ratio of TP to the sum of TP and FP.

(8) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{TP}{TP+FP}.$$

Recall is defined as the ratio of TP to the sum of TP and FN:

(9) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{TP}{TP+FN}.$$

One important indicator for assessing the effectiveness of object detection models, such the YOLO family, is average precision. The precision-recall curve, which shows the trade-off between precision and recall at different threshold values, is the source of this measure. The model’s accuracy in predicting items in a picture is summarized by AP.

AP 0.5 refers to the Average Precision calculated at a fixed IoU threshold of 0.5. The IoU is defined as the ratio of the area of overlap between the predicted bounding box and the ground truth bounding box to the area of their union, as shown in Eq. (10):

(10) $$\mathrm{I}\mathrm{o}\mathrm{U}=\frac{\mathrm{A}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{o}\mathrm{f}\mathrm{O}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{l}\mathrm{a}\mathrm{p}\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{t}\mathrm{h}\mathrm{b}\mathrm{o}\mathrm{x}}{\mathrm{A}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{o}\mathrm{f}\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{t}\mathrm{h}\mathrm{b}\mathrm{o}\mathrm{x}}.$$

In this case, if the IoU is greater than or equal to 0.5, the prediction is considered a true positive. The AP at this threshold, denoted as ( $A{P}_{0.5}$), is calculated as the average precision over all points in the precision-recall curve, as given by Eq. (11):

(11) $${\mathrm{A}\mathrm{P}}_{0.5}=\frac{1}{N}\sum _{n=1}^N\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}(n),$$where N represents the number of points on the precision-recall curve.

${\mathrm{A}\mathrm{P}}_{0.5-0.95}$ is a more comprehensive metric that averages the AP over multiple IoU thresholds, ranging from 0.5 to 0.95 in increments of 0.05. This broader evaluation provides a more rigorous assessment of the model’s performance, as it requires the model to achieve good performance across a spectrum of IoU values. The formula for this metric is shown in Eq. (12):

(12) $${\mathrm{A}\mathrm{P}}_{0.5-0.95}=\frac{1}{10}\sum _{t=0.5}^{0.95}\mathrm{A}\mathrm{P}(t),$$where $t$ denotes each IoU threshold from 0.5 to 0.95.

Illustrative numerical example

To clarify the practical differences between Non-Maximum Suppression (NMS), Weighted Box Fusion (WBF), and the proposed Dempster–Shafer Weighted Box Fusion (DS-WBF), we present a simple numerical example. Suppose the detection of the same object by the same YOLO model yields three bounding boxes with slightly different coordinates and confidence scores:

• Box A: $(x_1=100,y_1=150,x_2=200,y_2=250)$, confidence = 0.85

• Box B: $(x_1=102,y_1=148,x_2=198,y_2=252)$, confidence = 0.80

• Box C: $(x_1=105,y_1=155,x_2=205,y_2=255)$, confidence = 0.70.

All three boxes overlap significantly (IoU > 0.6), indicating they likely correspond to the same object.

NMS: NMS retains the box with the highest confidence score (Box A: 0.85) and discards the others, resulting in a single output box with coordinates of Box A and confidence = 0.85.

WBF: WBF merges the boxes by taking a weighted average of their coordinates, where the weights are proportional to the confidence scores:

$$x_{1,\mathrm{W}\mathrm{B}\mathrm{F}}=\frac{100\times 0.85+102\times 0.80+105\times 0.70}{0.85+0.80+0.70}\approx 102.14$$

(similarly for $y_1,x_2,y_2$). The resulting confidence is the weighted sum:

$${\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}}_{\mathrm{W}\mathrm{B}\mathrm{F}}=\frac{0.85+0.80+0.70}{3}\approx \mathrm{0.78.}$$

DS-WBF (Proposed): The proposed DS-WBF incorporates Dempster–Shafer theory to manage uncertainty and conflicting evidence. Confidence scores are treated as basic probability assignments (BPAs), and fusion is applied to compute a more robust belief value. Assuming the belief masses after Dempster–Shafer combination are:

$$m_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}=\mathrm{0.92.}$$

The final coordinates are computed similarly to WBF but weighted by the fused belief values rather than raw confidences, leading to improved localization and a higher, more reliable confidence score.