MSBOTS: a multiple small biological organism tracking system robust against non-ideal detection and segmentation conditions

Code of ethics

Ethics approval is not required when filming videos of larval organisms. Further, no chemicals were tested with the larval organisms being filmed. The adult zebrafish video analysed is from a publicly available online repository (Pérez-Escudero et al., 2014).

Organism detection and segmentation

In the background subtraction step of the proposed MSBOTS platform, an improved GMM (Zivkovic & Heijden, 2006) was chosen to estimate the stationary background due to its ability to extend the detection period (Wang et al., 2017a). This is particularly effective for addressing the imbalanced movement problem of small biological organisms that occur in time-lapse microscopic video frames. As reported by Liu et al. (2017), small organisms such as zebrafish larvae exhibit a mean proportion of activities less than 7.5% over time with a ‘bursty’ movement pattern of sudden swimming motion interspersed with mainly stationary periods.

After video background estimation, the moving objects can be initially classified as the foreground organisms. Due to the ‘bursty’ movement pattern of small organisms, a moving organism will be firstly represented by a new region cluster with a small weight, whose value will be gradually increasing if the region remains stationary over time. This new region will not be classified as the background until its weight value exceeds a threshold c_f, which is calculated as in Zivkovic & Heijden (2006). Thus, the detection period of a stationary organism can be extended for approximately log(1 − c_f)(1 − α) frames, where α denotes the constant factor shaping an exponential decay envelope introduced in the GMM work by Zivkovic & Heijden (2006). This enables the detection and tracking of organisms that stop at a position for a certain period of time before restarting their motion.

Furthermore, to adapt to background changes in microscopic videos over time, the adaptive GMM (Zivkovic & Heijden, 2006) adds flexibility when selecting the number of Gaussian components and their parameters. The approach by Zivkovic & Van Der Heijden (2004) and the Dirichlet criteria are applied, respectively, in model initialisation and selection of the number of Gaussian components. The Gaussian component parameters are then adaptively updated for each video frame to accurately represent background pixels, in contrast to traditional GMM models that apply one or a fixed number of components. The foreground objects are then obtained by the differentiation of the video frames from the corresponding estimated background. Furthermore, post-processes such as median filtering, morphological grayscale erosion, and size-based noise removal are applied to eliminate image distortion and scattered noise fragments from the foreground. Within these, the morphological grayscale erosion deploys a flat diamond-shaped structuring element to erode the obtained organism foreground image. Thus, the boundaries among organisms in very close proximity are further widened; thus, the estimated centroid positions of the detected regions are more accurate.

The detection and segmentation of MSBOTS enables the removal of stationary backgrounds (such as the organism container and labels/markers drawn on its surface) and is robust against unbalanced organism movement patterns (e.g., ‘bursty’ movement) and water impurities. Hence, unlike existing systems, MSBOTS is able to process videos under practical experimental imaging conditions.

Representing detected organisms

To represent the positions of the detected organisms in each video frame, the centroid locations of segmented foreground regions in Cartesian coordinates are used and stored in a vertical cell array matrix, as shown by the parallelogram series (indicating video frames) and Points{t} matrix in Fig. 2. In the Points{t} cell array, the first column stores the temporary identity, numbered from 1 to the number of detected organisms in each video frame, where n_obj(t) indicates the number of detected organisms in frame t. The second and third columns store the horizontal and vertical positions of each detected organism in x and y coordinates, respectively. This cell array matrix representation allows for varying element length (indicating the number of detected organisms in frame t), which can change frame-to-frame due to detection and segmentation errors.

Organism assignment between frames

The centroid locations of detected organisms are obtained in the segmentation module and represented frame-by-frame using a cell array for each video sequence as described in the previous section. However, the organism identities over video frames are still unknown. That is, which organism in the current frame corresponds to which organism in the following frame has not been mapped. In addition, there are still some remaining mis-detected organisms that have been classified into background regions or overlapped with other detected organisms. The data association module of MSBOTS thus builds tracking trajectories of individual organisms by mapping the detected organisms between successive pairs of frames, calculating the positions of mis-detected or occluded organisms, and bridging the theoretical locations back to their correct trajectories by adjusting the initial assignment results.

Position estimation for mis-detected and occluded organisms

Figure 3 illustrates the location computation for the overlapped organism P_i in the frame t + 1. The two points P_i (shown by blue dots) and P_j (shown by green dots) overlap with each other in the frame t + 1, and this overlapped point at time t + 1 is assigned to the object P_j in the initial mapping process. Since the point P_i in the frame t cannot find a target in the frame t + 1, and this point in the frame t + 2 cannot be assigned to a source point in frame t + 1, this point is classified as an unmatched source point in frame t, and an unmatched target point in frame t + 2, respectively. The position of the missing point due to occlusion (as shown by the red dot occluded by the green dot in Fig. 3) or mis-detection (for example, when the red dot is completely occluded by the green dot in Fig. 3) is calculated using the locations of the unmatched source point P_i(t) and unmatched target point P_i(t + 2). In this example, as shown in Fig. 3, the location of the missing point P_i in frame t + 1 is calculated by the median point between P_i(t) and P_i(t + 2).

When there are multiple unmatched pairs, the mapping from unmatched source points to the unmatched target points is also according to the extended Hungarian assignment algorithm (Bourgeois & Lassalle, 1971). In this step, the unmatched source points firstly search for possible correspondences in the following 2nd frame. If no assignment can be mapped, the search extends to the unmatched targets in the following 3rd frame. It was shown in Noss, Lorke & Niehaus (2013) that a trajectory fragment can be bridged to its subsequent tracking fragment so long as the frame separation is less than 6 video frames. Thus, the default search range in MSBOTS is from the 2nd to the 6th frame following the frame when the mis-detection and occlusions originally occur. The positions of missed organisms (x_miss, y_miss) are calculated by Eqs. (4) and (5) using the matched point pair from the the locations of unmatched source point (x_s, y_s) and an unmatched target point (x_t, y_t).

(4) $$x_{miss}=x_s+\frac{1}{j}\ast (x_t-x_s)$$

(5) $$y_{miss}=y_s+\frac{1}{j}\ast (y_t-y_s)$$

where j indicates the following j-th frame from the unmatched source point.

Small biological organism datasets

Microscopic time-lapse videos of three types of small organism models: zebrafish, Artemia and Daphnia were applied to evaluate the proposed MSBOTS platform. Both low frame rate videos (of 14 or 15 fps recorded by a AD7013MT Dino-Lite microscope) and high frame rate videos (captured by an UI-3360CP-C-HQ microscope) are tested in this work.

Wild zebrafish (Danio rerio) embryos were incubated at 28 °C in a petri dish filled with an E3 medium. Zebrafish larvae were obtained from the hatched embryos 5 days post-fertilization. Zebrafish larvae were transferred to round poly (methyl methacrylate) (PMMA) housing wells for shooting microscopic time-lapse videos. The zebrafish dataset consists of 10 video sequences with 3,056 frames in total: nine zebrafish larvae videos captured as prescribed here and one adult zebrafish video provided by Pérez-Escudero et al. (2014) from their publicly available online repository (‘Example video of 5 zebrafish’ at http://www.idtracker.es/download). The size of the zebrafish larvae video frames is 960 pixels × 1280 pixels, and the average zebrafish larvae size is around 1900 pixels. More details about the larvae zebrafish dataset and its generation can be found in our previous work (Wang et al., 2017b).

Cysts of the marine crustacean Artemia franciscana and freshwater Daphnia magna were hatched and cultured according to the Artoxkit-M and Daphtoxkit-F (MicroBioTests Inc., Gent, Belgium) standard operating protocols. Artemia franciscana were hatched in a petri dish filled with sea water (pH 8.0 ± 0.5) at 24 ± 0.5 °C under exposure to 3,000–4,000 lux light source for 30 h. Artemia were placed into a group of 10 in a miniaturised Lab-on-a-Chip (LOC) chamber as per Solis et al. (2015) when shooting videos with microfluidics infused at a flow rate of 5.25 mL/h. Five Daphnia magna neonates were randomly selected and transferred into a petri dish with the temperature maintained at 20.0 ± 0.5 °C. The Artemia franciscana and Daphnia magna dataset consists of 5 video sequences each with 4,802 frames and 4,804 frames in total, respectively.

Artemia franciscana microscopic videos containing five organisms and artifacts (bubbles of different sizes, video sequences 1–5), with Fig. 4B a microscopic video frame example of Artemia franciscana in 480 pixels × 640 pixels. The average size of an Artemia franciscana object is approximately 500 pixels. Daphnia magna microscopic videos containing 10 organisms and artifacts (bubbles and impurities of different sizes, video sequences 1–5), with Fig. 4C a microscopic video frame example of Daphnia magna in 480 pixels × 640 pixels. The average size of Daphnia magna tested is approximately 400 pixels. Both the Artemia and Daphnia videos were ordered randomly to test the flexibility and feasibility of the proposed system.

Tracking evaluation metrics

To objectively and quantitatively evaluate the tracking performance of the proposed MSBOTS platform, the widely utilised standard metric, Classification of Events, Activities and Relationships (CLEAR MOT) (Bernardin & Stiefelhagen, 2008), for Multiple Object Tracking (MOT) is employed in this paper. CLEAR MOT comprises two metrics: Multiple Object Tracking Precision (MOTP) as presented by Eq. (9) and Multiple Object Tracking Accuracy (MOTA) as shown by Eq. (10). MOTP measures the position precision of all segmented organisms compared to that of the manually labelled ground truth in every video frame, whereas MOTA estimates the individual trajectory accuracy (the ability to produce exactly one trajectory per organism with a consistent label over time).

(9) $$MOTP=\frac{\sum _{i,t}|P_{i,t}-G{T}_{i,t}|}{\sum _t{N}_t}$$

(10) $$MOTA=1-\frac{\sum _t(M_t+F{P}_t+SI{I}_t)}{\sum _t{g}_t}$$

where |P_i,t − GT_i,t| is the Euclidian distance between the estimated centorid position of the i-th detected organism P_i(t) in the frame t. Its position in the manually labeled ground truth is denoated as (GT_i,t) and N_t indicates the total number of segmented organisms in the ground truth in frame t. The smaller the MOTP value, the more precise the segmentation result.

MOTP measures the organism segmentation accuracy compared with the ground truth, whereas the MOTA metric emphasises the evaluation of the individual tracking trajectories, where M_t, FP_t, and SII_t indicate the number of missed detections, false positive detection (i.e., image noise fragments segmented as organisms), and the Swapping of Individual Identities (SII), respectively, in the frame t. And g_t means the total number of organisms detected in frame t. The ideal value of MOTA is 1, and the value decreases with the occurrence of detection errors and identity swapping. Comparing with the total number of detected organisms, the MOTA value will generate negative numbers (as shown in Fig. 5A) when the combination of detection errors and identity swapping is high, which implies low system reliability and the resultant tracking trajectories should be considered as unreliable.

Tracking accuracy evaluation

To evaluate the proposed tracking system, its tracking performance over the microscopic video dataset is compared with the well-known multiple object tracking platform idTracker (Pérez-Escudero et al., 2014), SimpleTracker (Bourgeois & Lassalle, 1971) (using the Kuhn-Munkres tracking algorithm for initial association and the nearest neighbourhood to directly connect trajectory fragments, without considering missed organisms), and the off-the-shelf commercial LoliTrack system (Závorka et al., 2017). Fig. 5 shows the estimated tracking accuracy measured by the MOTP and MOTA metrics, with Fig. 5A showing results from zebrafish videos, Fig. 5B showing Artemia videos, and Fig. 5C showing Daphnia videos.

It can be seen from Fig. 5A (the upper MOTP figure) that the positions of detected organisms using adult and larvae zebrafish videos consistently exhibit the smallest distance differences with the manually labelled ground truth positions amongst all the methods tested. The proposed MSBOTS approach resulted in overall smaller MOTP values of 0.92, 25.59, and 44.48 pixels (71.59% increase) compared to SimpleTracker, idTracker, and LoliTrack, respectively. The improved organism position detection results demonstrate the accuracy of the theoretical position estimation based on the organism detection and segmentation results using the adaptive GMM model and post-processing in the proposed MSBOTS system. Table S1 summarises the estimated tracking accuracy measured by MOTP and MOTA values across all the zebrafish video sequences.

All of the methods performed well for the position accuracy of detected organisms when the videos had a clear background, as shown by sequences 1–6. However, the location errors measured by MOTP for the detected organisms in the zebrafish videos compared to the ground-truth did not decrease as dramatically as LoliTrack or idTracker with increasingly complex video backgrounds, as shown by the MOTP values in sequences 7–10 in Fig. 5A. Accordingly, the proposed MSBOTS method still out-performed the existing approaches when taking into account mis-detection, false positive segmentation and identity swapping, with 31.20%, 63.01%, and 24.61% higher MOTA values than SimpleTracker, idTracker, and LoliTrack, respectively. This was mainly achieved by the ability to estimate the positions of the mis-detected or overlapped organisms using their neighbour position knowledge from the segmentation results in the proposed MSBOTS platform. In addition, the bridging of trajectory fragments in MSBOTS based on the extended Hungarian assignment algorithm (Bourgeois & Lassalle, 1971) when there are multiple unmatched trajectory fragments decreased the possibility of individual identity swapping, compared with SimpleTracker, which only used a distance metric by nearest neighbour algorithm (which in turn can generate identity swapping during the gap bridging process).

In addition, Figs. 5B and 5C show the tracking accuracy evaluation using Artemia franciscana and freshwater Daphnia magna videos, respectively, to test the generalisation of the proposed MSBOTS system on other small organisms with movement characteristics different to zebrafish larvae. Artemia display flexible movement and vary according to surrounding fluidics (Tyson, 1974; Williams, 1994), and Daphnia exhibits short, jerky hopping movement in water (Rottmann et al., 1992). The overall tracking accuracy performance of the proposed MSBOTS method is consistent for the tested videos on these two organism types. The detailed data on tracking accuracy measured by MOTP and MOTA values can be seen in Tables S2 and S3.

As can be seen from the MOTP values in Fig. 5B, the proposed method exhibits 47.68 pixels smaller standard deviation than the idTracker system, which illustrated the usability of the proposed system on Artemia microscopic videos and the enhanced ability on Artemia detection accuracy. Though idTracker can produce smaller organism position estimation errors (MOTP) as shown by sequences 3–5 in Fig. 5B, the mean MOTA value is 7.07%, and 6.44% less than the proposed MSBOTS and LoliTrack, respectively, which illustrates the existence of a similar detection problem when testing zebrafish larvae. That is, the organism is detected as background, and impurities as an organism, due to their small size differentiation and similar movement characteristics when the organism stops moving or the water impurities are stirred up by organism movement; this further causes identity confusion.

Figure 5C shows the overall tracking accuracy of the proposed MSBOTS system and idTracker consistently outperformed the other two systems under comparison, measured by MOTP and MOTA. However, the mean MOTA value of the proposed MSBOTS system is still 5.48% higher than idTracker. The generalised tracking accuracy of the proposed MSBOTS system is further illustrated by applying it to Daphnia video sequences with short, jerky hopping movement characteristics, compared with these existing systems.

Analysis of organism movement characteristics

To explore the capability of the proposed MSBOTS approach in analysing organism movement characteristics, this paper presents the estimation of movement velocity, acceleration and direction calculated from the individual tracking trajectories.

Figure 6 shows the movement acceleration analysis for each zebrafish video using the resultant tracking trajectories generated by the proposed MSBOTS. It was found that in Hinz & de Polavieja (2017), the interaction and movement of zebrafish larvae was very close to zero by 7 dpf. As shown in Fig. 6, in general the variation of movement speed (shown by the computed acceleration values) that is obviously visually perceptible to the human eye occurred only within approximately 10 s, as tested in the 10 adult and larvae zebrafish videos. This is due to the natural (anxious) response of zebrafish when turning on the imaging camera (Peng et al., 2016). The zebrafish movement speeds are more subtle from this point on, which is consistent with the zebrafish movement characteristics found in Peng et al. (2016); Hinz & de Polavieja (2017).

As organism movement speed and direction changes can provide insight into the interaction rules (Hinz & de Polavieja, 2017), Fig. 7 shows an example of the calculated velocity and movement direction results for each zebrafish video, respectively. The equal number of boxes with the number of zebrafish housing in each video show that all individuals were successfully assigned with one identity (which also illustrated the one-to-one organism mapping accuracy in the proposed MSBOTS system), and the median velocity (labelled by the red line inside each box) of each larvae indicates the consistent movement characteristic within the same housing well. Mean, minimum and maximum velocity values can also be easily obtained from the visual box plot shown in Fig. 7.

There is no specific speed requirement for the organisms that can be tracked by the MSBOTS platform. Both organisms travelling at high speed or average low speed as illustrated by Figs. 6 and 7A, respectively, can be accurately detected and tracked over time by the system. Thus, this system can be used for automatic tracking, comparison and analysis of small organisms in natural response or under the exposure of testing chemicals. The recommended lowest frame rate of time-lapse videos is 14 f/s as in the tested dataset from trial-and-error comparison.