FRVC: frame relevance based video compression for surveillance videos using deep learning methods

Object detection

Object detection (OD) is a technique in which the class of an object and its location is detected through bounding boxes in an image. After OD, researchers focus their attention on semantic segmentation, which aims to classify each pixel into a fixed set of categories without differentiating object instances. Semantic segmentation and object detection together form instance segmentation, which differentiates objects using pixel-level masking. These DL-based OD is divided into two categories which are summarized as follows:-

Video compression

Video compression is a process of reducing the bit count required to encode a video while maintaining its visual quality (Sayood, 2017). This is accomplished by identifying and exploiting the similarities between successive video frames. Standard videos typically run at 30 frames per second (FPS), where most of the content is identical across frames. The compression techniques search for residue information between the frames. Lossless video compression and lossy video compression are the two main types of video compression. lossless video compression attempts to shrink the size of video files without losing its video quality (Beach & Owen, 2018). While lossy video compression reduces video file sizes by removing some amount of data, typically the data that is less sensitive to the human visual system (Bull & Zhang, 2021). H.264 (Wiegand et al., 2003) and H.265 (Sullivan et al., 2012) are widely used lossy video compression algorithms that adapt the concept of a hybrid video coding framework. Here, input video data is split into frames, frames are divided into blocks and blocks into units, the largest unit is termed as control tree unit (CTU). CTU is also divided into the control unit (CU), CU into the prediction unit (PU), and at last PU into the transform unit (TU). These frames/blocks/units are compressed in a predefined manner termed as intra-frame prediction and in inter-frame prediction approaches. Earlier, compressed frames are used to compress or predict the next frame, respectively. Then predicted data are transformed, quantized, and entropy-coded to gain final coded bits. As predicted data is quantized, there might be a chance of losing some amount of data which may cause noise. To recover from this loss, two filtering strategies were suggested: (i) in-loop filtering (before predicting the next frame) and (ii) out-loop filtering (before output). In addition, to minimize the size of video data, the frames/blocks/units are down-sampled before compression and up-sampled afterward. H.265 and H.264 adopted this traditional procedure to perform compression while researchers tried to apply the concept of neural networks to the tools of this framework for e.g., intra-prediction, inter-prediction, filtering, sampling, and entropy coding is performed using DL approaches and achieved remarkable success in compression (Liu et al., 2020).

Recently, researchers focused on improving video compression by designing end-to-end trainable architectures that jointly optimize key components such as motion estimation, residual encoding, and entropy coding (Rippel, Nair & Lew, 2019). Lu et al. (2019) performs video compression in an end-to-end manner, learning-based optical flow estimation is used to acquire motion information and to reconstruct the current frames. Following this, two neural networks designed auto-encoders are used to compress both the associated motion and residual data. All of these components are collectively trained using a unified loss function. Agustsson et al. (2020) introduces a simplified approach using scale-space flow, a generalized motion model with a scale parameter to better handle disocclusions and fast motion. The proposed low-latency model, which avoids B-frames and pre-trained optical flow networks, achieves competitive or superior rate-distortion performance compared to state-of-the-art learned video compression methods, while significantly reduces architectural and training complexity. Though DL algorithms represent a promising avenue for advancing video compression techniques, there is potential for advancement in the field of surveillance video compression.

Surveillance video compression

This sub-section gives a review of video compression approaches applied to surveillance video. Lu & Xu (2019) presents a deep compression algorithm for apron surveillance. Researchers used the Fast-R-CNN (Girshick, 2015) OD technique to predict the object in apron surveillance and then the object is cropped according to its coordinates and saved on disk in linked list format. In this way, the foreground and background parts of images are differentiated with illumination and brightness information. While performing decompression, using coordinate information cropped objects placed at their position in background images and this approach efficiently solves the space issue. Ghamsarian et al. (2020) perform compression on a cataract surgery video using semantic segmentation approaches and perform categorization of a frame into active and idle frames. Later, active frames of videos are compressed with different quantization parameters under five different scenarios from an ophthalmologist’s point of view. Wu et al. (2020) perform compression on the foreground and background images in parallel using an end-to-end neural network approach. For foreground compression, motion estimation with residual encoding is used, and to share the background with neighboring frames the interpolation method is applied. In the article (Mohod, Agrawal & Madaan, 2024), object detection-based surveillance compression (ODSC) technique identifies objects within surveillance videos. This method utilizes various YOLO modules to distinguish between significant and non-significant frames. ODSC model used four surveillance videos for testing purposes and achieved 96% compression rate that outperforms other methods. To address error accumulation in inter-frame prediction for surveillance video compression, Zhao et al. (2024) proposes a Temporal Adaptive enhancement method for surveillance video compression (TALC) with Forward and Backward modules. These modules use adaptive selection and feature enhancement blocks to reduce distortion. TALC improves compression performance across various learned frameworks without altering their core architecture. UVCNet is an end-to-end unsupervised neural video compression framework that separates foreground and background online using temporal correlation. It applies residual coding for dynamic foreground and specialized compression for static background. UVCNet optimizes bitrate by leveraging scene dynamics and achieves a 2.11 dB PSNR improvement over H.265 on surveillance video datasets (Zhao et al., 2023).

Dataset preparation

Relevance frame classification

In the relevance frame classification module, we applied YOLOv9 from a one-phase detector and Mask R-CNN from a two-phase detector to identify relevant and irrelevant frames of surveillance videos. Unlike conventional OD approaches that detect all objects indiscriminately, relevance-based detection aims to optimize video processing tasks such as compression. Recent methods such as semantic-assisted object cluster (SOC) leverage temporal and textual cues to group relevant objects across frames, improving temporal coherence and detection performance (Li et al., 2023). Another notable approach, TransVOD, employs a spatial-temporal transformer architecture that integrates information across frames for more accurate object localization and classification (He et al., 2023). The proposed FRVC framework incorporates this concept of relevance by identifying objects that are not only present but also contextually important in terms of human activity and motion. Compared to earlier methods, our relevance approach focuses on retaining semantically meaningful content, which leads to enhanced compression efficiency without losing critical surveillance information using OD. The details architecture of YOLOv9 and Mask R-CNN detectors are explained as follows.

YOLOv9

Wang, Yeh & Liao (2024) presents you only look once version 9 i.e., YOLOv9 to tackle the issue of losing information in the deeper network. Unlike other YOLO, its architecture is divided into three parts: the backbone, neck, and head. The backbone of YOLOv9 incorporates advanced techniques such as programmable gradient information (PGI) and generalized efficient layer aggregation networks (GELAN). PGI optimizes gradient flow and improves training stability, efficiency, and feature learning in deep networks. While GELAN combines two neural network architectures, CSPNet (Wang et al., 2020) and ELAN (Wang, Liao & Yeh, 2022) to optimize feature extraction using a lightweight architecture. Figure 3 denotes the architecture diagram of YOLOv9. It also incorporates gradient path planning for better performance. The neck of YOLOv9 utilizes a PanNet to integrate multi-scale features and improve spatial context and localization accuracy. As we go deeper into the network, good quality semantic features are extracted but lacking in spatial features which are present at the early layers of the network. In order to merge both features, FPN uses top-down and bottom-up paths. While PanNet used the lateral connection to shorten the information path and focused on enhancing the high-resolution feature maps with detailed information. It consists of 1 × 1 convolutions that transform feature maps from the backbone into a common channel dimension. These transformed feature maps are then added element-wise with the higher-resolution feature maps from the bottom-up path. This step is crucial for aligning features across scales. At last, YOLO layers perform detection at multiple scales. This network places anchor boxes on the feature map generated by the preceding layer and produces a vector containing the target object’s category probability, object score, and the bounding box’s position. YOLOv9 also employs an asymmetric downsampling (A DOWN) block instead of max pooling, to preserve essential spatial information and to reduce the size of the feature map.

Mask R-CNN

He et al. (2017) introduced a novel, intuitive, flexible, and simple network to perform instance segmentation named as Mask R-CNN. The Mask R-CNN architecture embodies the essence of faster R-CNN while extending its capabilities to do instance-level segmentation. Hence, Mask R-CNN performs object categorization, bounding box regression, and pixel-level segmentation simultaneously. Figure 4 gives architecture details of Mask R-CNN network. Mask R-CNN used ResNet-50/ResNet-101 architecture as a backbone to extract high-quality features from images (He et al., 2016). The input image is passed through the backbone network, and a series of convolutional layers extract low-level and high-level features, which represent various aspects of the image, from edges to abstract object features. To address the issue of vanishing gradients, ResNet uses residual block connection which includes a “skip connection” to bypass one or more layers. This residual block consists of CNN, BN, skip connection and rectified linear unit (ReLU) activation function. Similar to YOLOv9, Mask R-CNN uses FPN with the ResNet model to extract fined and coarse features at various scales. After feature extraction, Mask R-CNN generates RPN that examines the features generated from the backbone and proposes the potential regions of images that might contain objects. For these regions, the RPN produces objectness scores and precise bounding box coordinates using anchor-box and non-max suppression approaches. After obtaining region proposals from the RPN, the next step is RoI Align. RoI Align is crucial for ensuring that the features within each region proposal are consistently aligned and resized to a fixed spatial dimension. In faster R-CNN, RoI pooling is used to extract fixed-size features from the proposals. However, in Mask R-CNN, the RoI-align operation is used to extract accurate and aligned features from the proposals, which addresses the issue of spatial misalignment between the feature map and the original image. RoI-align uses bilinear interpolation to retrieve feature values from the original feature map at non-integer locations rather than quantizing the RoI boundaries to the grid cells. Hence, using RoI alignment the loss of spatial information that occurs in RoI pooling is recoverable in Mask R-CNN and generates a binary mask for each object in an image. At last, the head component is responsible for making predictions and is divided into two parallel branches: object classification and mask prediction. The final output for each detected object includes the predicted bounding box coordinates, the assigned class label, and an instance segmentation mask, enabling both object detection and detailed instance segmentation. Figures 5 and 6 denote the relevant frames and irrelevant frames of surveillance video using Mask R-CNN and YOLOv9 module, respectively.

Compression module

In the compression module of the FRVC framework, processing is performed on the relevant frames of ATM surveillance video which is identified using the relevance frame classification module. While the irrelevant frames are discarded from the network. On relevant frames, we determine the similarity index between frames across various threshold values and further categorize these frames into primary frames and similar frames. Primary frames are defined as frames that exhibit difference when compared with their consecutive frames. If consecutive frames are identical, they are categorized as similar frames and the first dissimilar frame is marked as the next primary frame. For instance, the 1^st frame is considered as the primary frame and it is compared with its consecutive frames. If the initial frame is similar to the 2^nd, 3^rd, 4^th and 5^th frames but dissimilar to the 6^th frame, then 1^st and 6^th frame are considered as primary frames, while 2^nd, 3^rd, 4^th and 5^th frames are designated as similar frames. As the 6^th frame is dissimilar to the first frame, it is subsequently identified as the next primary frame and the process continues until all relevant frames exhaust. We determine the similarity between frames at various threshold values ranging from 100% to 90% and save the count of the similar frames corresponding to the primary frame in a data structure. Lastly, the video is constructed using relevant primary frames, at the original FPS. Equation (1) gives the formula to predict the similarity between frames (Wang et al., 2004). The Structural Similarity Index (SSIM) is a composite measure that evaluates the similarity between two images based on three components: luminance, contrast, and structure. The luminance component quantifies the similarity in average brightness, computed using the means of pixel intensities. The contrast component measures variability in the images, expressed through standard deviations. The structure component evaluates the correlation between the two images using covariance. These components are combined multiplicatively to yield the overall SSIM index.

(1) $$\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(x,y)=\frac{\left(L(x,y)\cdot c_1\right)\cdot \left(L^{\prime }(x,y)\cdot c_2\right)}{\left(L(x,y)+L^{\prime }(x,y)+c_1\right)\cdot \left(c_2+c_3\right)}.$$

In the above equation, the variables represent various components used to quantify the SSIM between two images, x and y. Here’s an explanation of these variables:

• (i)

  x and y (Images): x and y represent the pixel values of the two images. The formula assesses the similarity between these two images.

• (ii)

  L(x, y) (Luminance Comparison): The L(x, y) term represents the luminance comparison between the images. It measures the similarity in terms of brightness and contrast between x and y.

• (iii)

  L’(x, y) (Contrast Comparison): The L’(x, y) term is the contrast comparison component. It quantifies the structure and variations in the contrast between the two images.

• (iv)

  c1, c2, and c3 (Constants): These are constants introduced in the SSIM formula to stabilize the division and prevent division by zero errors. They are typically small positive values that are added to the denominator to ensure numerical stability.

FRVC algorithm

To implement the FRVC model, we used Sypder version-5 (Python 3.9) on intel 11th Generation, i7 Processor. The required configuration of the system is 16 GB RAM and 512 GB SSD on the Windows platform with NVIDIA GeForce GPU. Algorithm 1 demonstrates the step-by-step execution of the proposed FRVC model:-

• (1)

  The algorithm begins by determining the FPS of the input video. Later, FRVC extracts the frames from the video.

• (2)

  Then various variable lists are initialized to store the data. For e.g., relevantFrames is an empty list created to store frames that are considered relevant. PrimaryFrames is an empty list to save primary frames, and similarFrames is an empty list to store similar frames. NextPrimary is initialized with a value of 1 and will be used to track the next primary frame. similarityThreshold is an initial value used as a reference for measuring the similarity of frames at various threshold values (100%, 98%, 96%, 94%, 92%, 90%).

• (3)

  The algorithm goes through all the frames of the video. and apply two object detection algorithms: Mask R-CNN and YOLOv9, to identify objects within the frame. It checks if the frame is relevant based on the detected objects. If a frame contains relevant objects (e.g., person), it adds the frame to the relevant frames list. If not, it discards the frame as it is not considered relevant.

• (4)

  For each relevant frame in the relevant frames list, the algorithm compares the frame’s similarity with the next frame in the list. If the frame’s similarity with the next frame is above the similarity threshold, it is classified as a similar frame and added to the similar frames list. If the similarity is below the threshold, the frame is classified as a primary frame and added to the primary frames list. Additionally, the next primary variable is updated to keep track of the current primary frame. The count of the similar frame corresponding to the primary frame is maintained in an Excel file.

• (5)

  Finally, the algorithm constructs the compressed surveillance video using the relevant primary frames at the original FPS. This step creates a new video that retains important frames while removing less significant ones, effectively compressing the video. After the construction of the compressed video, the data structure is stored on the disk.

The sequence flow diagram of the FRVC model is shown in Fig. 7. Here, the framework is divided into the following steps:-

• (i)

  In the initial phase, namely “dataset preparation”, the process begins with the collection of surveillance videos, and this video is further split into frames. Subsequently, annotations are applied to these frames.

• (ii)

  In the second phase i.e., “relevance frame classification” We trained the one-phase YOLOv9 and the two-phase Mask R-CNN on the FRVC dataset. Subsequently, we employed these trained models to analyze the remaining ATM surveillance video as part of our testing process and predict the relevant and irrelevant frames of surveillance video. Depending upon the evaluation metrics and time required to identify the relevant and irrelevant frames, the best model for OD is determined.

• (iii)

  In the final stage, known as “Video compression”, relevant frames are taken as input, and depending upon similarity at various threshold values primary and similar frames are categorized. In the end, the compressed video is constructed using relevant primary frames at the original video’s FPS value.

Decompression

To perform decompression of the video, the list (data structure) stored on the disk is retrieved. During compression, the count of similar frames corresponding to primary frames are stored in the list. This list guides us to reconstruct original video using Algorithm 2 that demonstrates the flow of the decompression module. Firstly, the algorithm initializes essential variables and sets up the decompressed video. Subsequently, it designates the first primary frame as the starting point for decompression and appends it to the decompressed video. The algorithm then enters a loop that iterates through the frames stored in the data structure. Within this loop, each primary frame encountered determines the count of similar frames related to that primary frame. Through a nested loop, the algorithm appends each of these similar frames to the decompressed video. After processing the similar frames, the algorithm progresses to the next primary frame and includes it in the decompressed video. This iterative process continues until all frames in the data structure have been processed. Ultimately, the algorithm concludes by returning the fully reconstructed decompressed video, effectively restoring the original content based on the categorization of frames established during compression in the FRVC framework.

Despite attaining the highest compression of 96.3%, our decompression algorithm ensures the retrieval of 100% of relevant data. The pre-processing operation is not applied to the frames of surveillance video, an inherent characteristic emerges where the frames exhibit a 100% similarity to each other when the threshold value is set to 1. However, as the threshold values decrease from 98% to 90%, the observed similarity between frames progressively diminishes. This decline in similarity can be attributed to the comparison process, wherein the first frame is systematically evaluated against the second or third frame. As a consequence, the stringent threshold values lead to a reduction in the permissible dissimilarity between consecutive frames, resulting in the recorded similarity indices decreasing from 100% to 96% over this threshold range.

Evaluation metrics used in object detection approach

To identify relevant and irrelevant frames of surveillance video, a DL-based OD approach is used, and to evaluate these techniques following parameters are applied:-

Evaluation and analysis of OD Module

This subsection gives a detailed analysis of the implemented relevance frame classification module. The FRVC algorithm is applied to seven ATM surveillance videos where Table 2 provides a comprehensive overview of the key characteristics of surveillance video files. It systematically lists pertinent details for each video, including the video number (No.), period of capture (day or night), FPS, resolution, duration, and size. The initial five videos comprise day-time surveillance video, whereas the last two videos feature night-time surveillance. To train YOLOv9 and Mask R-CNN architecture, we consider four different possible combinations of surveillance dataset in a ratio 90–10%, 85–15%, 80–20%, and 75–25% for training-testing and observed their performance in terms of accuracy, speed, recall, and precision. Table 3 gives an idea about the parameter specification required to train YOLOv9 and the Mask R-CNN framework. As the detection accuracy of the two-phase detector is higher than the one-phase detector, we used 40 epochs with batch size 16 to train YOLOv9 while only 30 epochs with batch size 2 to train the Mask R-CNN network. During training, Mask R-CNN uses various loss functions with adam optimizer and a learning rate of 0.01. The loss function consists of RPN loss, classification loss, regression loss, and mask loss. While the loss function in YOLOv9 is also a combination of objectness loss, regression loss, classification loss, balanced loss, and weighted loss. The gradients of this loss are used to update the network’s parameters during backpropagation.

After an extensive experiment, it is observed that 80–20% ratio gives promising performance and achieves the highest accuracy. Table 4 offers a nuanced understanding of the performance characteristics of the YOLOv9 and Mask R-CNN network, considering key metrics and different data distribution scenarios The YOLOv9 module, achieved the highest accuracy of 99.6% using a split ratio of 80–20% while for Mask R-CNN it is 99.2%. The highest precision of 95.6% is achieved using YOLov9 and 93.8% for Mask R-CNN using 80–20% ratio. Moving to recall, a measure of a module capability to identify all relevant instances, YOLOv5s exhibits consistent improvements, reaching 99.3% and Mask R-CNN showcases varying recall values, with the highest 99.5% achieved at 80–20 ratio. Figure 8 shows the performance of YOLOv9 in terms of evaluation metrics. Derived from these results, an examination reveals that customized YOLOv9 exhibits promising results for the surveillance dataset as compared to Mask R-CNN. Notably, the detection accuracy of the one-phase YOLOv9 model surpasses the two-phase Mask R-CNN. Furthermore, it is observed that Mask R-CNN requires a maximum amount of time (probably twice or thrice) for the detection of relevant and irrelevant frames within the video as compared to YOLOv9. Figure 9 shows the graphical comparison of OD modules on seven different videos. Hence, the output generated through the YOLOv9 module is employed for subsequent compression tasks.

Table 5 represents the count of relevant and irrelevant frames using the YOLOv9 network on seven different tested videos and the time taken by each video to predict the frames. The video2 has the highest number of total frames i.e., 35,990, of which only 5,725 are relevant, while 30,265 are irrelevant and require 12,579 s for detection. Video3 has a notable 13,477 frames, with the highest 9,577 as relevant frames and 3,900 are detected as irrelevant which indicates the maximum transaction activity video. Conversely, Video5 has 12,769 total frames, but only 60 are relevant, which suggests minimal activity. The analysis reveals that longer processing times are required for videos with high frame counts. Hence, Table 5 underscores the importance of YOLOv9 in automated video frame classification and its relevance for applications like surveillance and storage compression.

Evaluation and analysis of compression module

In the compression module, the output of the YOLOv9 module is used to conduct further compression processes. In the compression process, the similarity index between the relevant frames of surveillance video is determined at various threshold values starting from 100% to 90% and the frames are splits into primary frames and similar frames. It is observed that, at 100% threshold value all relevant frames are primary frames, hence there is no further division of frames, while this splitting starts from 98% threshold value.

Table 6 systematically displays the maximum compression achieved at a 90% threshold value and provides insights into the efficiency of the compression process in specific scenarios. In this article, compression efficiency is determined in terms of compression ratio and percentage of compression achieved. Compression ratio is a quantitative metric used to evaluate the efficiency of a video compression technique. It is defined as the ratio of the uncompressed (original) video size to the compressed video size. A higher compression ratio signifies a more efficient compression algorithm, as it reflects a greater reduction in data volume required to represent the original content. While the percentage of compression achieved represents the percentage of size reduction after compression. Each video captures transactional activities, which focuses on the presence and actions of individuals within the surveillance video. We analyze videos across three distinct scenarios. Figure 10 represents sample frames of the three scenarios of videos. Scenario-I refers to a situation when single individuals enter an ATM room and engage in transactional activities. Specifically, Video-1, Video-2, and Video-3 belong to Scenario-I and these demonstrate the highest compressions within the dataset, reaching 96.3%, 93.0%, and 92.5%, respectively, when individuals are detected. In Scenario-II, the video depicts a timeframe where no human presence is detected. Specifically, Video-5 corresponds to Scenario-II. For Video-5, out of the 12,769 frames, only 60 frames are identified as relevant, achieving an impressive compression rate of 99.9% at a threshold value of 90%. This implies that the entire video is effectively recognized based on a small subset of frames. In fact, at a 100% threshold value, the compression rate is still substantial, reaching 98.9%. While scenario-III entails videos-4, video-6 and video-7, where two individuals jointly enter the surveillance room and engage in transactional activities. In this scenario, the frames exhibit lower similarity, resulting in a larger proportion being considered as primary frames. Consequently, these videos achieve lower compressions of 49.3%, 61.3%, and 63.3%, respectively, compared to other scenarios. The percentage of video compression is determined using Eq. (6). Table 6 provides compression ratios achieved by the FRVC framework across different surveillance scenarios, indicating how many times the original videos were compressed. Scenario-II (no human presence) achieved the highest compression ratio of 1,369.52, reflecting minimal scene changes. Scenario-I (single-person activity) achieved ratios between 13.34 and 27.16, while Scenario-III (multi-person activity) showed lower ratios around 2 to 3 due to increased scene complexity. These results highlight the influence of video content on compression efficiency. It is essential to note that, while conducting video compression across various threshold values, a deliberate effort is made to uphold the video’s quality. This entails maintaining the values of FPS and frame resolution consistently, aligning them with the original specifications. This strategic approach is designed to prioritize the preservation of essential visual characteristics throughout the compression process and ensure that the perceptual quality of the video remains uncompromised. Figure 11 provides a visual representation of frame count between different threshold values for each video. For example, video-1.avi contains a total of 26,996 frames out of which 2,736 frames are recognized as relevant frames using the YOLOv9 network which is further divided into 2,491 primary frames and 245 similar frames at a 98% threshold, resulting in a video size of 104.2 MB. Similarly, at a 96% threshold, there are 1,709 primary frames, 1,025 similar frames, and a reduced video size of 60.5 MB. The count of primary frames further decreases to 1,157, with 1,579 similar frames, yielding a compressed video size of 35.5 MB at a 94% threshold. At a 92% threshold, 780 primary frames, 1,956 similar frames, and a video size of 26.4 MB are observed. Finally, at a 90% threshold, there are 539 primary frames, 2,197 similar frames, and a video size of 19.4 MB. Table 7 details the compressed video size at various threshold values, while Fig. 12 graphically represents the size of the video. It is evident that as the similarity value between frames decreases, the compression size increases. The highest compression is achieved at a 90% threshold, while the minimum compression is observed at a 100% threshold.

(6) $$Compression(\mathrm{\%})=\frac{\mathrm{O}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{S}\mathrm{i}\mathrm{z}\mathrm{e}-\mathrm{C}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{e}\mathrm{d}\mathrm{S}\mathrm{i}\mathrm{z}\mathrm{e}}{\mathrm{O}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{S}\mathrm{i}\mathrm{z}\mathrm{e}}\times 100.$$

Table 7 also serves as a quantitative representation of the impact of similarity threshold values on video compression. It allows for a detailed examination of how variations in these thresholds influence the compression ratios for different videos. At a 100% threshold, a minimum 66.2% average compression reached is achieved. For the 98% threshold, the average compression is 70.1%, while at 96%, it is recorded as 73.2%. The average compression for a 92% threshold is 80.6%, and the highest compression of 85.1% is achieved at a 90% threshold. Notably, the average compression increases as the similarity threshold decreases, the observed trends in compression sizes provide valuable insights into the relationship between similarity threshold values and the efficiency of the compression algorithm, which can be crucial for optimizing video storage and transmission in surveillance systems. Based on the above analysis, it is observed that the most significant compression is observed in Scenario I, where a lone individual engages in transactional activities. Conversely, Scenario II, characterized by the absence of relevant parts, is instrumental in optimizing storage space. Notably, the highest compression achieved is 96.3%, with an average compression of 85.1% at a 90% threshold value. The observed trend underscores that compression increases as the similarity between frames diminishes. Figure 13 visually encapsulates this relationship through a Pareto chart, providing a comprehensive overview of the outcomes across the seven tested videos. The Pareto chart discernibly demonstrates the inherent trade-off between prioritizing similarity and compression. It demonstrates that while increasing similarity thresholds reduces individual compression rates, the cumulative compression remains significant. Decision-makers should carefully select similarity thresholds based on the desired trade-off between data accuracy and compression efficiency. For applications where data accuracy is critical, higher thresholds should be preferred despite lower compression. However, for storage-intensive applications where efficiency is prioritized, lower thresholds may be more suitable. Comprehensively, these findings contribute to the optimization of video storage and transmission within surveillance systems, that underscore the delicate balance between similarity thresholds, compression efficiency, and visual fidelity. The scientific approach employed ensures that key video attributes, such as FPS and resolution, are preserved throughout the compression process, which prioritizes the retention of essential visual information.

State-of-the-art works comparison

Table 8 presents a comparative analysis of the proposed FRVC framework with three state-of-the-art video compression approaches tailored for surveillance and medical video scenarios. Each method employs different strategies for video frame compression and relevance detection, and the table highlights key parameters such as the compression approach, object detection (OD) techniques used, frame resolution, FPS, compression ratio, and type of compression.

The first approach, by Panneerselvam et al. (2023), leverages a combination of CNN and GAN to perform pixel-level compression. Their approach focuses on eliminating duplicate frames and captures minor changes between frames using GAN and long short-term memory (LSTM) networks for recording the altered frames. This method achieves a compression ratio of up to 50.71% and processes video frames at 25 FPS. However, it does not incorporate any explicit object detection module, and frame resolution details are not specified. The second approach, by Ghamsarian et al. (2020), is designed for cataract surgery videos and uses Mask-RCNN for semantic segmentation and classification of video frames based on their relevance. This approach focuses on reducing bitrate by assigning less importance to non-critical frames, thereby achieving a compression ratio of up to 63%. The video frames are processed at 20 FPS, but, similar to the previous method, the exact frame resolution is not mentioned. Lu & Xu (2019) present a deep compression technique applied to apron surveillance videos. Their method employs Faster-RCNN to distinguish between moving and stationary objects and selectively compress frames based on this object separation. It achieves a higher compression ratio of up to 88% with a frame resolution of 1,920 $\times$ 1,080 and a frame rate of 25 FPS. This method stores extracted objects and background separately to facilitate compression.

The proposed FRVC framework outperforms all the aforementioned methods in terms of compression efficiency. It uses the latest YOLOv9 object detection approach, enabling more precise identification of relevant frames in ATM surveillance videos. The framework achieves an impressive compression ratio of up to 96.3% in Scenario-I and 99.9% in Scenario-II, both at a high resolution of 1,920 $\times$ 1,080 pixels. Although the frame rate after compression is slightly lower at 15 FPS, this is a reasonable trade-off given the significant improvement in compression ratio. Like the other approaches, the FRVC framework employs lossy compression. Overall, the table illustrates that the FRVC framework provides a superior balance of compression ratio, frame quality, and object detection accuracy by incorporating advanced detection methods and relevance classification, thus demonstrating a clear advantage over existing state-of-the-art solutions.