Backlight and dim space object detection based on a novel event camera

Event cameras

An event camera is a new type of neuromorphic visual sensor inspired by the biological visual system, and differs from frame-based visual sensor in the principle of operation. The exposure time of a frame camera is fixed, and the exposure is repeated even if there’s no change in illumination on a pixel. In contrast, each pixel of an event camera independently detects changes in illumination, generating asynchronous event data, which includes timestamps, pixel addresses, and the polarity of illumination change.

Event cameras have independent asynchronous pixels that report triggered local illumination changes. The circuit for a single pixel is shown in Fig. 1A. For each pixel, whenever the illumination change reaches a set threshold, it outputs an asynchronous event, as shown in Fig. 1B, where $V$ represents the logarithm of the illumination intensity and t represents the time. If the log-compressed light of the pixel increases by a fixed amount, the pixel sends an ON event asynchronously. If the log-compressed light of the pixel increases by a fixed amount, the pixel sends an OFF event asynchronously. This event-based asynchronous data format is called address event representation (AER) (Delbruck et al., 2010) and is used to simulate the transmission of neural signals in biological vision systems, as shown in Fig. 1A. With this representation, information is continuously transmitted and processed, and the communication bandwidth is only occupied by the pixel that triggers the event.

Event-based object detection

Recent advancements in event-based object detectors have addressed deficiencies of traditional optical cameras in complex visual scenarios. Current methodologies can be categorized into event images (Chen, 2018; Cannici et al., 2019; Iacono, Weber & Glover, 2018; Li et al., 2019) and event volumes (Hu, Delbruck & Liu, 2020; Perot et al., 2020). Event images convert asynchronous events into binary or grayscale images for use in traditional frame-based detectors. Event volumes integrate event data into continuous volumetric format. Both types generate 2D, image-like representations compatible with frame-based systems, but struggle to fully exploit spatio-temporal characteristics of asynchronous events.

A critical review of existing methods (Messikommer et al., 2020; Iacono, Weber & Glover, 2018; Li et al., 2019; Hu, Delbruck & Liu, 2020; Perot et al., 2020) indicates that these approaches predominantly utilize a simplistic strategy by deploying a feed-forward, frame-based detection model independently for each event image, which fails to capitalize on the rich temporal cues and motion information. This limitation has been primarily due to the absence of large-scale, continuous event datasets that are amenable to supervised learning applications, before the release of the Gen1 automotive dataset (Tournemire et al., 2020) and the 1Mpx detection dataset (Perot et al., 2020). Drawing on principles from video object detection (Han et al., 2016; Chen, Yu & Wu, 2020; Zhu & Liu, 2018), which make use of temporal information across multiple adjacent frames, this article proposes a novel method for event-based space object detection, which distinctively processes asynchronous events and harnesses rich temporal cues through an innovative asynchronous convolutional memory architecture, aiming to substantially improve the utilization of temporal data in event-based detection systems.

Event representation

A fundamental issue in the emerging field of event cameras is the optimal extraction of spatio-temporal representations from asynchronous events for specific applications (Gehrig et al., 2019; Roy, Jaiswal & Panda, 2019; Gallego et al., 2022). The literature categorizes existing methods into four primary approaches: image-like conversions, handcrafted descriptors, spiking neural networks (SNNs), and deep neural networks (DNNs). Initial approaches involved transforming asynchronous events into image-like planes, either within a fixed temporal window or based on a constant event count (Maqueda et al., 2018; Zhu et al., 2019). These were employed in various computer vision tasks such as image reconstruction (Rebecq et al., 2021), object classification (Amir et al., 2017; Wang et al., 2019), detection (Jiang et al., 2019; Li et al., 2019), and tracking (Gehrig et al., 2020b). Furthermore, some studies have developed efficient spatiotemporal feature descriptors from asynchronous events, including temporal surfaces (Lagorce et al., 2017; Sironi et al., 2018) and DART (Ramesh et al., 2019). However, these descriptors, often utilized in corner and edge detection, are not only time-intensive to design but also heavily dependent on the dynamics of the moving objects involved. SNNs represent another approach, where end-to-end learning of representations is theoretically possible (Shrestha & Orchard, 2018; Paredes-Valles, Scheper & Croon, 2020; Neftci, Mostafa & Zenke, 2019). Despite this, SNNs have yet to achieve performance on par with DNNs and have predominantly been applied to classification tasks, not extending to numerical regression challenges. In recent developments, there has been a significant push towards creating end-to-end learning frameworks using DNNs. Innovations such as EST (Gehrig et al., 2019), Matrix-LSTM (Messikommer et al., 2020), and Sparse-Conv (Cannici et al., 2020) have been specifically designed to directly process asynchronous events, achieving cutting-edge results. These advances highlight the increasing efficacy and applicability of DNNs in harnessing the full potential of event-based data for complex computational tasks.

Long short-term memory (LSTM)

Recurrent neural networks (RNNs) (Yu et al., 2019) and their variants, including long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU) (Cho et al., 2014), are well-known for handling sequential inputs and modeling long-term dependencies. Recently, these networks have been integrated with convolutional neural networks (CNNs) (LeCun, Bengio & Hinton, 2015) to create recurrent convolutional architectures, particularly for video-related tasks (Zhu & Liu, 2018; Cruz & Bernardino, 2019; Li, Liu & Wang, 2019; Lai et al., 2020; Wang et al., 2021). These architectures aim to leverage both spatial and temporal memory.

Several recurrent convolutional architectures have been developed for video object detection (Zhu & Liu, 2018; Cruz & Bernardino, 2019). For example, Cruz & Bernardino (2019) used convolutional LSTM to capture temporal features in maritime airborne video object detection. Similarly, Zhu & Liu (2018) introduced a bottleneck LSTM layer that reduces computational demands compared to traditional LSTM configurations. Despite these advancements, these systems often suffer from high computational complexity, which can be a limiting factor.

To address these challenges, this article introduces a convolutional spatiotemporal memory module ConvLSTM designed to efficiently extract temporal features. This module stacks multiple ConvLSTM layers to construct a robust detection framework suitable for a feed-forward event object detector leveraging asynchronous events. Our architecture aims to diminish computational costs while enhancing performance and preserving the capacity to maintain long-term dependencies effectively.

Synthetic events

An overview of event modeling using event cameras is provided. Early research by Kaiser et al. (2016) developed a basic method of event generation by applying a threshold to image frame differences. Pix2NVS (Bi & Andreopoulos, 2017) calculates per-pixel brightness in conventional video frames to generate events. The first accurate event generators are in references (Mueggler et al., 2017) and Li et al. (2018), using high frame rate rendering and linear interpolation of intensity signals. Rebecq, Gehrig & Scaramuzza (2018) proposed an adaptive sampling mechanism, adjusting to maximum displacement between frames. The generative models in Rebecq, Gehrig & Scaramuzza (2018) and Mueggler et al. (2017) are underpinned by formalizations established in earlier studies (Lichtsteiner, Posch & Delbruck, 2008; Gallego et al., 2015, 2017). These foundations significantly contributed to advancing and understanding event-based modeling, highlighting potential applications.

Overview of the ACMNet

Due to the particularity of the space scene, the target is usually far away and small, and lacks significant appearance features under extreme lighting conditions. It is difficult to separate the target from the background by appearance features alone, and a large number of motion features are hidden in the time series information, which is easy to be ignored. Therefore, an extraction method combining appearance features and timing features is proposed in this article, and based on this method, an asynchronous convolutional memory network (ACMNet) for spatial object detection is constructed. The network framework is shown in Fig. 2. Taking asynchronous event streams as input, the kernel function is converted into EST voxel grid to retain timing features. After the appearance features are extracted in the feedforward feature extraction network, the BiSA-ConvLSTM module is constructed to memory time series features, and multi-scale feature fusion and regression prediction algorithm are used to output accurate target detection results.

The algorithm process is as follows:

EST voxel grid

Under the conditions of backlight and dim light, the space object usually lacks significant appearance features, so it is very important to extract the motion information of the object. These motion information are fused into the temporal information of the event stream, so it is necessary to transform the event stream data into a representation that can effectively retain the timing characteristics. The unique representation of EST voxel grid can effectively preserve the event and its temporal characteristics with high time resolution, which is especially critical for detecting the backlight and faint targets with weak appearance characteristics. Therefore, the characterization method of EST voxel grid was adopted in this article.

Event cameras have pixels that are independent and respond to changes in the continuous log brightness signal $L\left(u,t\right)$. Event cameras mark a time stamp with a microsecond-level resolution and output asynchronous events, as shown in Fig. 3. This example from the dynamic vision sensor shows how a high-speed stimulus generates a sparse, asynchronous digital stream of events which rapidly signify changes in scene reflectance (Delbruck et al., 2010).

An event $e_k=\left(x_k,y_k,p_k,t_k\right)$ is triggered when the magnitude of the log brightness at the pixel $u_k={\left(x_k,y_k\right)}^{\mathrm{T}}$ and time $t_k$ has changed by more than a threshold $C$ since the last event at the same pixel.

(1) $$\mathrm{\Delta }L\left(u,t_k\right)=L\left(u,t_k\right)-L\left(u,t_k-\mathrm{\Delta }{t}_k\right)\ge p_{k}C$$where, $\mathrm{\Delta }{t}_k$ is the time since the last triggered event, $p_k\in \left\{-1,+1\right\}$ is the sign of the change, also called the polarity of the event.

In a given temporal interval, the event camera will trigger many events:

(2) $$E={\left\{e_i\right\}}_{i=1}^N={\left\{\left(x_i,y_i,t_i,p_i\right)\right\}}_{i=1}^N$$

Due to the asynchronous characteristic, events are represented as a set. To use events in combination with a convolutional neural network, it is necessary to convert the event set into a grid-like representation. Events represent point-sets in a four-dimensional manifold spanned by the x and y spatial coordinates, time, and polarity. This point-set can be summarized by the event field, inspired by Rebecq, Gehrig & Scaramuzza (2018) and Gallego et al. (2022):

(3) $$S\left(x,y,t\right)=\sum _{e_i\in E}{p}_i\mathit{\delta }\left(x-x_i,y-y_i\right)\mathit{\delta }\left(t-t_i\right)$$

This representation replaces each event by a Dirac pulse $\delta$ in the space-time manifold. We convolve it with an exponential kernel function $k\left(u,t\right)$, and the convolved signal thus becomes:

(4) $$\begin{array}{rl}\left(k\ast S\right)\left(x,y,t\right)& =\sum _{e_i\in E}{p}_{i}k\left(x-x_i,y-y_i,t-t_i\right)\\ k\left(x,y,t\right)& =\mathit{\delta }\left(x,y\right){\displaystyle \frac{1}{\tau }\exp \left(-t/\tau \right)}\end{array}$$

After kernel convolutions, a grid representation of events can be realized by sampling the convolved signal at regular intervals:

(5) $$V\left[x_w,y_h,t_b\right]=\left(k\ast S\right)\left(x_w,y_h,t_b\right)=\sum _{e_i\in E}{p}_{i}k\left(x_w-x_i,y_h-y_i,t_b-t_i\right)$$

$\left[x_w,y_h,t_b\right]$ is the spatiotemporal coordinates of the EST voxel grid, $x_w\in \left\{0,1,\cdots ,W-1\right\}$, $y_h\in \left\{0,1,\cdots ,H-1\right\}$, $t_b\in \left\{t_0,t_0+\mathrm{\Delta }t,\cdots ,t_0+Bin\mathrm{\Delta }t\right\}$, where $t_0$ is the first timestamp, $\mathrm{\Delta }t$ is the bin size, and $Bin$ is the number of temporal bins.

The representation of EST voxel grid effectively retains the high temporal resolution and spatiotemporal characteristic of the event, especially for small and faint objects that lack obvious spatial characteristics, this representation retains the characteristic of the time dimension, which is beneficial to subsequent object detection.

Feed-forward feature extraction network (CSP-DarkNet)

CSPDarkNet includes a focus structure and a CSP structure. Focus structure is a convolutional neural network layer for feature extraction, which is used as the first convolutional layer in the network to down-sample the input feature map, compress and combine the information in the feature map, to extract a higher-level feature representation, and reduce the amount of computation and parameters. The original feature map is input into a focus structure, sliced, and turned into a 320 × 320 × 12 feature map. The slicing operation is shown in Fig. 4.

The core idea of the CSP structure is that the input feature graph is divided into two parts, one part is processed by a small convolutional network (called a sub-network) and the other part is directly processed in the next layer. The two feature maps are then spliced together as input for the next layer. It can significantly reduce the parameters and computation of the network, and improve the efficiency of feature extraction, to speed up the training and inference of the model.

In this article, CSP-DarkNet is used as the feed-forward feature extraction network to extract object features at different spatial scales. The input of CSP-DarkNet is an image with a size of 640 × 640 × 3. To transfer CSP-DarkNet to the task in this article, the number of input channels is modified to be W × H × bin, and the value of batch size is B, which is set in training and testing to adapt to the EST voxel grid as network input. The output of the network is $x$ a W × H × bin dimensional feed-forward feature tensor.

Asynchronous event streams contain rich temporal information, but existing event-based object detectors (Messikommer et al., 2020; Chen, 2018; Iacono, Weber & Glover, 2018; Li et al., 2019; Jiang et al., 2018) suffer from the loss of rich temporal information and motion information. Since the feature of the object is not obvious during remote observation, it is necessary to continuously observe and fully analyze and extract temporal information. Therefore, we effectively retain temporal information by transferring the representation of the EST voxel grid and propose to add ConvLSTM, a convolutional spatiotemporal memory module, to CSP-DarkNet to further extract temporal features. The feature extraction network of ACMNet consists of a feed-forward feature extraction network and a convolutional spatio-temporal memory module. Its structure is shown in Fig. 5.

Convolutional spatio-temporal memory module (ConvLSTM)

Asynchronous event stream is a kind of data format, that contains rich temporal information, especially when dealing with dynamic scenarios. However, the existing event-based object detectors are faced with the problem of loss of temporal and motion information when dealing with such data, especially under complex lighting conditions. When observing long-distance objects, the features are not obvious, this problem is more prominent. To solve the above problems, it is necessary not only to observe the event stream continuously but also to analyze and extract the temporal information deeply. The solution in this article converts the event stream into a voxel grid representation, preserving its temporal information. Then, these temporal features are further extracted in the feature extraction network.

A long short-term memory network (LSTM) is a model used to process time sequence data, which can extract time domain features from arbitrary sequence data (Hu, Delbruck & Liu, 2020). FC-LSTM is an LSTM structure based on a fully connected layer, which better captures the spatial connection relationship by adding a snooping mechanism. However, because it contains too much data, there is a lot of redundancy.

In this article, we extend the conventional FC-LSTM idea to ConvLSTM by replacing the input-to-state and state-to-state calculations in FC-LSTM with convolution operations and forming a detection structure by superimposing multiple ConvLSTM layers to extract temporal correlation features.

ACMNet contains $n$ sub-network structures, expressed as $A=\left\{a_1,a_2,\cdots ,a_n\right\}$, where the feed-forward feature extraction network CSP-DarkNet includes $m$ sub-network structures, and its output is a feed-forward feature tensor $x_t$, and it can be expressed as:

(6) $$x_t=\left(a_1\bullet a_2\bullet \cdots \bullet a_m\right)V_t$$where $\bullet$ denotes a connection between two adjacent layers in the feed-forward feature extraction network CSP-DarkNet.

The CspLayer layer of CSP-DarkNet is connected to ConvLSTM. The internal structure of ConvLSTM is shown in Fig. 6. The equation is as follows:

(7) $$\begin{array}{rl}& i_t=\sigma \left(W_{xi}\ast x_t+W_{hi}\ast h_{t-1}+W_{ci}\ast c_{t-1}+b_i\right)\\ & f_t=\sigma \left(W_{xf}\ast x_t+W_{hf}{h}_{t-1}+W_{cf}\ast c_{t-1}+b_f\right)\\ & c_t=f_t\circ c_{t-1}+i_t\circ \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(W_{xc}\ast x_t+W_{hc}\ast h_{t-1}+b_c\right)\\ & o_t=\sigma \left(W_{xo}\ast x_t+W_{ho}\ast h_{t-1}+W_{co}\circ c_t+b_o\right)\\ & h_t=o_t\circ \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(c_t\right)\end{array}$$where $\ast$ denotes convolution operation, $\circ$ denotes Hadamard product.

For a temporal bin $V_t$, ConvLSTM $L$ takes the feed-forward feature tensor $x_t$ and the memory state $h_{t-1}$ from previous temporal bins as the input, then outputs the current state $h_t$ and a spatiotemporal feature tensor $x_t^{+}$, and it can be described as:

(8) $$\left(x_t^{+},h_t\right)=L\left(x_t,h_{t-1}\right)$$

The last part of ACMNet is the spatiotemporal feature tensor passing through the neck and head modules to output the final detection result, described as:

(9) $$B_t=\left(a_{m+1}\bullet a_{m+2}\bullet \cdots \bullet a_n\right)x_t^{+}$$

The convolutional spatiotemporal memory module retains the object’s temporal information by combining a synchronization grid. LSTM is used to improve the processing ability of the module for time series data, which effectively improves the performance of the object detector in processing asynchronous event streams. Especially under complex illumination and remote observation conditions, this module can significantly improve the recognition ability of object features, thus improving the detection accuracy of the overall network.

Datasets

Since event cameras have not been applied in space so far, there is a lack of data captured by event cameras in space scenes. The space video data captured by spaceborne optical cameras are rich and contain a large amount of space object information. We use the video generation event model to synthesize existing space video data into event data and build a space synthetic event dataset, namely Event_DVS_space7. Space video data are selected to contain a large number of backlight, dim light, and other scenes where the use of conventional optical cameras is limited, to synthesize challenging event data to complete the verification of our proposed method.

Space synthetic event dataset

The space synthetic event dataset we built has the following characteristics:

(1) The event generation model adopts the principle of the dynamic vision camera (DVS) model. The time sampling rate is $1.6\times {10}^8$, dynamic range is 120 dB, latency is 1 μs, and the pixel resolution is $640\times 360$. The synthetic event dataset is called Event_DVS_space7.

(2) The method of building the dataset in this article directly relies on the video data recorded in the real space scenes, so a large number of real space video data, such as backlight and dim light, are selected to generate synthetic event data in the corresponding scene, which provides a basis for the verification of proposed method.

(3) Seven kinds of common space objects are selected to build datasets, including the sun, moon, stars, earth, astronauts, spacecraft, point objects such as small-sized space debris, and long-distance spacecraft. The proportion of various object quantities in Event_DVS_space7 is as follows: sun 7%, moon 4%, stars 6%, earth 31%, astronauts 3%, spacecraft 12%, point objects 37%.

In addition, the building of Event_DVS_space7 also fully considers the working mode of the event camera, the observation distance, and the number of objects and light intensity. Some of the scenes are shown in Fig. 7. Figures 7A and 7B respectively show the frame images and event images of the spacecraft during earth observation and sky survey observation. When the object is close to the detector, it presents surface object characteristics that the outline information can be identified. During long-distance detection, the objects often appear as point objects, including small-sized space debris and long-distance spacecraft, as shown in Figs. 7C and 7D. Figures 7E and 7F show the single object and multiple objects observed by two kinds of cameras respectively. Figures 7G and 7H show the frame image and event image when the light changes from dim light to backlight during the rising of the sun from the horizon, which can highlight the performance of the event camera under different lighting conditions.

We evaluate our proposed method on the Event_DVS_space7 dataset with a resolution of 640 × 360 and containing 1,830 samples. Event_DVS_space7 dataset is divided into training, validation, and test sets according to a proportion of 8:1:1.

Implementation details and metrics

Implementation Details. We train the proposed model using the PyTorch toolbox on the Ubuntu 20.04 LTS operating system. We use the Adam optimizer with a fixed learning rate of 0.0001. We train the model for approximately 500 iterations and 8 batches to obtain the final result. The training is performed on a PC with an i5-13400F processor @4.6 GHz, 16 GB RAM, and an NVIDIA RTX 3060Ti GPU.

Metrics. Three commonly used metrics, average precision (AP), mean average precision (mAP) and runtime, are used to evaluate the performance of ACMNet in this article. Runtime refers to the time consumed from inputting an image to outputting the final result, including pre-processing time (such as image normalization), network front-pass time, and post-processing (such as non-maximum suppression). The calculation formulas of AP and mAP are as follows:

(11) $$\mathrm{A}\mathrm{P}={\int }_0^{1}PdR$$

(12) $$\mathrm{m}\mathrm{A}\mathrm{P}={\displaystyle \frac{\sum _{i=1}^{N}A{P}_i}{N}}$$where $P$ and $R$ represent precision and recall, respectively; $N$ represents the number of detection categories.

Effective experiments

Quantitative evaluation, visualization evaluation, and experimental analysis will be highlighted to see why and how our ACMNet works as follows.

Visualization evaluation

Some representative visualization results on Event_DVS_space7 dataset are illustrated in Fig. 8. Note that, our ACMNet achieves the best performance against state-of-the-art methods including integrating event stream into event images SSD (Liu et al., 2016), YOLOV5 (Shariff et al., 2022). Figures 8A–8C are the multi-object scene of sky survey observation, the short-distance object scene of earth observation, and the tiny object scene of dim light observation, respectively. ACMNet shows better detection performance: from the ground truth of Fig. 8A, it can be seen that there are seven objects in the field of view, while the SSD network has one missed detection. As can be seen from Fig. 8B, both SSD and YOLOV5 networks can achieve over 90% accuracy in surface object detection, which is better than point object detection. It can be seen from Figs. 8A–8C that ACMNet shows better detection performance, especially for the effective detection of tiny objects in dim light conditions, while cannot be captured by SSD and YOLOV5. This is because ACMNet benefits from the representation of events and the convolutional spatio-temporal memory module, which can obtain effective temporal information from event streams. ACMNet is robust and adaptable to object detection with different working modes, detection distances, the number of objects, and light intensity.

Ablation study

Beyond the effective test, we next conduct ablation tests to take a deep look at the impact of each design choice and parameter settings for our ACMNet, and more details are presented in Table 2 and Fig. 9. As shown in Table 2, we further analyze how much each component to the ultimate performance. Notably, our two baselines, namely network I and II, utilize VoxelGrid and ConvLSTM respectively that, and consistently achieve higher performance than the benchmark (Shariff et al., 2022). More precisely, by introducing ConvLSTM module, the detection accuracy of network II increases by 3.6% based on the benchmark, and ACMNet increased by 8.6% compared to network I, indicating that the ConvLSTM module plays a positive role in improving detection accuracy. By introducing the VoxelGrid module, network I obtained 4.2% higher than the benchmark, and ACMNet improved mAP by 9.1% compared to Network II, verifying that the event stream representation method of EST voxel grid is more beneficial to improve the detection performance than the conversion to the event image. In addition, the last row of Table 2 illustrates that the detection speed of our ACMNet, introducing VoxelGrid and ConvLSTM, is almost comparable in contrast to the benchmark (Shariff et al., 2022).

Figure 9 presents the results in continuous event stream detection of the baseline network and ACMNet. Among them, Figs. 9A and 9B represent the classical scenes for earth observation and sky survey observation respectively. It can be observed from the Fig. 9 that ACMNet outperforms the baseline network in terms of detection performance. This further validates the effectiveness of these two modules in space object detection. Additionally, the baseline network without the convolutional spatiotemporal memory module (network I) performs better than other object detectors (Gehrig et al., 2019; Messikommer et al., 2020; Jiang et al., 2019; Shariff et al., 2022; Ren et al., 2017; Liu et al., 2016), demonstrating both high detection accuracy and fast detection speed.

The ablation study results validate the positive role of the VoxelGrid and ConvLSTM modules in the network, and their impact on detection speed is minimal. This part of the work enables a deeper understanding of the model's working principle, offering valuable guidance on model design and optimization.

Scalability test

This section studies two contents: Firstly, the performance of event cameras and conventional optical cameras are compared for backlight and dim space object detection. In addition, the influence of aggregation time of event streams on detection performance is studied.

(1) Three classical detection frameworks (Faster R-CNN (Ren et al., 2017) , SSD (Liu et al., 2016) , YOLOV5 (Shariff et al., 2022)) are selected for backlight and dim space object detection, and compared by using frame images and event images. Table 3 shows the detection results on Event_DVS_space7 dataset, among which YOLOV5 has a slightly higher detection accuracy. Compared with the event-based network, the frame-based network has better detection performance, mainly because the frame image resolution is much higher than the event camera and contains rich RGB information. There are a large number of earth observation, surface target and other scenes in the data set, and the frame-based network has more advantages in the detection of such scenes. But it is difficult to catch the target in backlight and dim light.

The visual detection results of frame images and event images are shown in Fig. 10. Figures 10A and 10B show the surface object and point object scenes of earth observation respectively. The network based on frame images has better detection performance. The main reason is that frame images have higher resolution than event images and contain rich RGB information. Due to the asynchronous characteristics of the event camera, the object may be submerged in the background noise, resulting in missed detection. However, the frame-based detection makes it difficult to capture the object under the backlight and dim light, as shown in Figs. 10C and 10D, which show scenes of backlight and dim light respectively. Event-based detection has obvious advantages in these scenes, especially when the frame image is almost invisible to the tiny object in dim light.

To sum up, frame-based object detection is more suitable for scenes with complex backgrounds, slow or relatively static motion, earth observation, etc., while event-based object detection is more suitable in backlight, dim light, and tiny object scenes. Therefore, the joint detection frameworks based on event image and frame image are more beneficial to the research in the space field. When event streams need static information to complete high-precision identification, frame-based object detection is the main method; However, when the frame image fails to capture objects in challenging scenes, event-based object detection is the main method.

It is worth noting that the event-based object detection in this section takes the representation of event images and the YOLOV5 framework as examples. Other event representations and detection frameworks (such as Faster R-CNN, SSD, etc.) can be used as alternatives, so this part of the work provides a general interface for the input of asynchronous events.

(2) Temporal aggregation Size: As shown in the Table 4, we investigate the effect of the aggregation time of the input event stream on the final detection performance. The aggregation time was selected as 25, 50, 75, 100, 125, 150, 200 and 300 ms. In the data set, the corresponding mAP differences were increased by 3.6%, 10.6% and 18.4%, respectively, compared with the aggregation time of 25 ms. 18.6%, 19.5% and 19.8%. Obviously, the accuracy tends to continue to improve with the increase of the polymerization size over time. This shows that from the size of the continuous event stream, the longer the aggregation time, the more time information is stored in the spatiotemporal memory module, thus obtaining better detection performance. Therefore, it is reasonable to set the polymerization time to 100 ms for good performance. This precision and speed trade-off is useful in some scenarios where higher precision detection performance is required.