Background subtraction for night videos

Weber contrast descriptor

As an example, in Fig. 2, area (3) contains a dim foreground object which has a small color deviation from the background samples, and the threshold to distinguish it from background should be small. In area (2), the pixel changes in the foreground object is much greater, so the threshold should be larger.

To address this problem, we propose a more effective pixel-wise descriptor named the Weber contrast descriptor, which is a term borrowed from the Weber-Fechner law (Fechner, 1860; Fechner, 1966). This law indicates that visual systems have different sensitivity to changes in lightness when lightening is different (Fechner, 1860; Fechner, 1966). For example, in dim areas, our visual perception system can catch subtle change but in bright areas, it is more difficult to perceive same changes. With this idea, we define our Weber contrast descriptor in a simplified version as follows: (1)${\displaystyle W=\frac{\Delta I}{I},}$

where ΔI is the actual change, namely, the deviation of intensity between background and current frame, and I is intensity of current frame. Equation (1) shows that objects in lighted area have relatively low Weber contrast values while objects in dim areas have relatively high values. Therefore, the detection of dim foreground objects is more effective, and the impact of lighting is not too great with the improved descriptor. The Weber contrast features are shown in Fig. 3B.

Textural features

Local pattern enhancement

The Weber contrast descriptor requires the elimination of strong lighting influences as well as another light detection method. Hence, we propose another light detection mechanism for further elimination of light.

A lighted area gradually fades in background with a smooth decrease of intensity. On the contrary, the foreground region of interests (FROI) like cars and pedestrians, have obvious silhouettes in common, which means that we can distinguish FROI and lights from changing areas using this edge features.

We seek to obtain more local textual information to better describe the silhouette of FROI. Inspired by local binary patterns, we design a local pattern that includes more detailed information, as shown in Fig. 4 to resolve the problem. With this pattern, we consider the differences between all the marginal pixels and the center pixel, and then summarize the differences. We use the summation as a new descriptor, to indicate local textural features at pixel x, y. Mathematically, it is defined as ${\sum }_{i=0}^{8s-1}|L_i-C_{x,y}|$, where s is the stride of pattern we have used in Fig. 4. In order to satisfy the condition to detect dim objects without strong lighting, the calculation of A(x, y) is modified as: (2)${\displaystyle A\left(x,y\right)=\sum _{i=0}^{8s-1}|L_i-C_{x,y}|\ast \frac{I_d}{max\left(C_{x,y},T_c\right)},}$ where I_d is intensity degree, meaning that when the brightness is less than I_d, the textural features should be enhanced. Otherwise, in high color intensity areas, the textural features will be suppressed by the multiplication of a small coefficient. T_c corresponds to threshold of camouflage. In low intensity areas, A(x, y) is very large because of the additional term, so we set the threshold T_c at 75 by default to avoid such large value. We need to normalize A(x, y) to combine the textural features with the Weber contrast features, which is shown in the following equation: (3)${\displaystyle A_{norm}\left(x,y\right)=\frac{A\left(x,y\right)}{255},}$ where s is stride with default value two. This is the Local Pattern Enhancement (LPE) approach.

Light detection unit

We introduce detection of the light in night videos by combining an intrinsic attribute of color and saturation. There are many alternative representations of the RGB color model according to different applications, HSV (Hue-Saturation-Value) or HSL (Hue-Saturation-Lightness) (Joblove & Greenberg, 1978). Color space is one of such representations. The lighted areas in night videos are similar to natural light, which has a low saturation. Thus, the lighted areas can be separated from other areas in HSV/HSL color space by using a specific threshold on saturation.

In our case we only consider the terms saturation (S) and value (V). The definition of saturation in HSV and HSL is only slightly different in that the colors with maximal saturation locate at lightness 0.5 in HSL, while they locate at value 1 in HSV. Light sources typically have low saturation but high value in HSV, such as headlights belong to foreground objects. Coincidentally, in HSL they have a relative high saturation just like other non-lighting objects. Thus, HSL is more suitable for our approach. Figure 5 shows the saturation maps calculated in HSV and HSL color space.

White or almost-gray objects have low saturation and the direct use of saturation will cause more false negatives. According to the Dark Channel Prior (He, Sun & Tang, 2011), in normal objects, there is at least one low intensity channel, but in lighted objects, characteristic are more like sky patches. Therefore, we can improve the saturation S_cor as follows: (5)${\displaystyle S_{cor}\left(x,y\right)=\left\{\begin{array}{ccc}\hfill & \hfill 1\hfill & \text{if}\frac{maxRGB\left(x,y\right)-minRGB\left(x,y\right)}{N_{all}}<T_d\hfill \\ \hfill & \hfill \frac{S\left(x,y\right)}{minRGB\left(x,y\right)}\hfill & \text{otherwise}\hfill \end{array}\right.}$

where S represents saturation in HSL, N_all is the number of pixels of an input image, and T_d is an upper boundary, minRGB(x, y) and maxRGB(x, y) are the minimum value and the maximum value of three channels (R, G, and B) for pixel position (x, y). In HSL biconical space, the closer the color vector is to the axis, the smaller saturation it has, and the saturation corresponding to the gray color is zero. Therefore, the light detection unit works only when the RGB information is not close to the gray value. When images taken at night are very close to gray images, the saturation method is not useful and we set S_cor(x, y) = 1 in this case.

Background match and classification

We have introduced the necessary features in our approach. The matching and background classification strategy are also important and must be combined to work effectively. One of the most important combined features in Fig. 1 is the effective feature which was calculated by following equation: (6)${\displaystyle F_{effective}=W\ast S_{cor}+A_{final}^2,}$ where W, S_cor and A_final are the matrices corresponding to Weber contrast, corrected saturation and final utilized textural features respectively. The value of each pixel is at the corresponding position in the matrix. In fact, the lighted areas masked by light detection units belong to other motion objects and were not classified as FROI. Thus, their pixels should not be updated into the background model, and the model update must be separated from the classification. Here the additional enhanced features F_enhanced for model update was defined as: (7)${\displaystyle F_{enhanced}=W+A_{final}^2.}$ The process of classification is similar to SuBSENSE (St-Charles, Bilodeau & Bergevin, 2015), this process is presented in Eq. (8), (8)${\displaystyle S_{mask}=\left\{\begin{array}{ccc}\hfill 1& \hfill \hfill & \text{if}\#\left\{F_{effective,i}<R,\forall i\right\}<\#min\hfill \\ \hfill 0& \hfill \hfill & \text{otherwise}\hfill \end{array}\right.}$ (9)${\displaystyle M_{mask}=\left\{\begin{array}{ccc}\hfill 1& \hfill \hfill & \text{if}\#\left\{F_{enhanced,i}<R,\forall i\right\}<\#min\hfill \\ \hfill 0& \hfill \hfill & \text{otherwise}\hfill \end{array}\right.}$ where S_mask is a segment mask for background segmentation while M_mask is a motion mask for model update. R is the matching threshold. The symbol #{.} means the number of true elements. #min is defined as the minimum number of matches, and the parameter #min = 2 is also used.

Background Model Update

To obtain a dynamic background model, once the pixel of the current detected frame is close to the background samples, this pixel should be updated into the background model. At the same time, the image with the farthest distribution in the background model should be replaced. We set a weight for each image to decide which image should be replaced or not. The similar images are allocated by higher weights, and vice versa. The definition of the weight for samples is as following equations: (10)${\displaystyle S_{mask}=\left\{\begin{array}{ccc}\hfill W_i+2& \hfill \hfill & \text{if}|I_{i,s}-I_{cur}|<T_{lower}\hfill \\ \hfill W_i+1& \hfill \hfill & \text{if}|I_{i,s}-I_{cur}|<T_{upper}\hfill \\ \hfill W_i-1& \hfill \hfill & \text{otherwise}\hfill \end{array}\right.}$ where I_i,s is intensity of the ith image, while I_cur is the intensity of current frame. T_upper corresponds to threshold R_color in Jiang & Lu (2018) whose value is 23, while T_lower was given as 10 to increase the weight of highly similar images. Such updating strategy can ensure the high correlation of inner images. The lowest weight of image was replaced.

Additionally, to eliminate ghost objects, we adopt the policy of random updating neighboring pixels with time subsampling (St-Charles, Bilodeau & Bergevin, 2015). It should be noted that the updating policy of update is only suitable for neighboring pixels, and update of the current pixel is triggered when M_mask = 1.

Parameter Initialization

We initialized a few parameters, and the threshold of the Weber contrast was one of the most important. The Weber contrast descriptor is highly sensitive in dark areas, while in some scenes such dark objects belong to FROI but in other scenes are noises such as shadows. Thus, a proper strategy to balance it is to use the global threshold rather than local threshold for parameter R, because the global threshold considers mainly changing areas. However, the global threshold means that each image has a separate threshold, and it is unfair to compare with other algorithms which are without parameter tuning. Thus, local threshold has been adopted in our approach by choose R as follows: (11)${\displaystyle R=C_{coe}-\frac{I_{s\text{\_}norm}}{2}}$ where I_{s_norm} is the normalized intensity of samples, and C_coe is correction coefficient. In the evaluation, we use C_coe = 0.51 in tramStation and fluidHighway and C_coe = 0.41 in streetCornerAtNight, winterStreet, busyBoulvard and bridgeEntry. R is typically bound within the range of [0.12, 0.4]. It means that with the decrease of color intensity, the local threshold R will be increased. Using Eq. (11) we can suppress the high sensitivity in dark areas.

Evaluation

We evaluated our approach using the seven measures and their results are shown in Table 1. The F-measure plays an important role in the evaluation of overall performance. There are results of comparisons with other state-of-art algorithms as shown in Table 2 using series of video frames with ground-truth information. Our approach shows a competitive performance for night videos with dim objects. To illustrate the advantage of our approach, images representing typical conditions such as lighting and dim objects are shown in Fig. 6. For some frames, our approach does not perform as good as other approaches, which is mainly due to the lack in shadow detections. Our approach is implemented in python. This program has been optimized through justintime(jit) based on numba, running with Intel core i5 at 1.6GHz and the processing speed is three frames per second.