Masked timesteps in spiking neural networks for efficient time series classification

Spiking neural networks for time series classification

Regarded as the third generation of neural networks (Maass, 1997), SNNs differ from traditional ML and DL models by emphasizing biological plausibility and energy efficiency. Furthermore, the intrinsic ability of spiking neurons to capture temporal dependencies makes SNNs particularly well-suited for TSC applications in embedded systems, wearable devices, and remote monitoring.

Current methodologies for TSC-SNN are divided into two categories, as illustrated in Fig. 1: reservoir-based methods and deep learning-based methods. The reservoir-based methods include Reservoir Computing (RC) and Echo State Networks (ESN). The deep learning-based methods are further divided into ANN2SNN (ANN to SNN) conversions (Rueckauer et al., 2017) and models trained from scratch (Wu et al., 2019; Sam et al., 2023).

Reservoir computing, specifically ESN, which uses a particular Recurrent Neural Network (RNN) as the reservoir, simplifies training by requiring only adjustments to the weights from the reservoir to the output layer. The dynamic properties of the reservoir efficiently capture the temporal dependencies of the input series. Peng et al. (2024) developed two RC variants for multivariate TSC tasks. Dey et al. (2022) proposed a spike-based reservoir architecture for TSC with two spike-encoding methods to classify vibration time series datasets, and demonstrated that temporal encoding is more effective for TSC than rate encoding. Zhang, Zhu & Li (2018) proposed the Spiking Echo State CNN (SES-CNN), which uses a specialized temporal encoding technique to transform time series data into multichannel spike train signals. The encoded features are extracted using the reservoir computing approach with spiking neurons. The spikes are then converted into 2D mapping images, facilitating classification through a CNN (Zhang, Zhu & Li, 2018).

Although RC, particularly ESN, has advantages in scenarios that are not too complex and require adaptive training, advances in deep learning have overshadowed some of the benefits that RC once offered.

An alternative method to TSC-SNNs uses deep learning techniques, including models converted from ANN as well as those trained from scratch. Yan, Zhou & Wong (2021) introduced a method for energy-efficient ECG classification using an ANN2SNN-based TSC-SNN model. However, the ANN2SNN method requires prolonged timesteps for conversion and imposes constraints such as setting biases to zero and absorbing batch normalization layers (Su et al., 2023). Moreover, the information extraction process is still conducted by well-trained ANNs, which contradicts the biological plausibility of SNNs.

To address this issue, directly-trained SNNs have gained popularity. Fang, Shrestha & Qiu (2020) first applied SNNs to multivariate TSC tasks. They proposed an encoding scheme that converts time series into sparse spatio-temporal spike patterns. Then an iterative SNN model was derived from the Spike Response Model (SRM), enabling the use of Back Propagation Through Time (BPTT) within SNNs (Fang, Shrestha & Qiu, 2020). Gautam & Singh (2020) proposed a hybrid CSNN architecture that uses a differentiable spike generated by the Soft-LIF model to facilitate backpropagation. However, using Soft-LIF compromises the most distinctive feature of SNNs, the binary spike. Gaurav, Stewart & Yi (2023) proposed two TSC-SNN models, inspired by the theories of reservoir computing networks and Legendre Memory Units: one based on RC and another trained using surrogate gradient methods. These models demonstrate the potential of SNNs in temporal data for performance and energy efficiency by nonlinearly decoding the linearly extracted features through spiking neurons (Gaurav, Stewart & Yi, 2023).

Though previous TSC-SNN models have shown some progress, they still encounter certain deficiencies. Some models rely heavily on convolutional layers for feature extraction and consist of only single spike neuron layers, without fully replacing all ReLU units with spiking neurons. Others require complex preprocessing of the original data or the use of sophisticated pre-training encoders. Moreover, some models need multiple sets of reservoir parameters, show limited performance, or have been tested only on specific datasets.

Temporal redundancy in spiking neural networks

Recent studies have pointed out the temporal redundancy in SNNs. Kim et al. (2023) discovered a phenomenon called Temporal Information Concentration (TIC) during the training of SNNs, which indicates the concentration of key information in the early timesteps. As training stabilizes, the sensitivity of the model to new information decreases, leading to great temporal redundancy in later training stages (Kim et al., 2023). Li, Jones & Furber (2023) employed Dynamic Confidence to optimize the latency of inference by dynamically determining when to terminate inference, thereby reducing timesteps.

Several studies are dedicated to overcoming temporal redundancy by reducing timesteps. Li et al. (2024) proposed the Spiking Early-Exit Neural Networks (SEENN), treating the number of timesteps as a variable conditioned on the input samples and using two methods to determine the exit timestep. They also proposed Dynamic Timestep SNN (DT-SNN), which dynamically determines the number of timesteps during inference (Li et al., 2023c). Ding et al. (2024) proposed Shrinking SNN (SSNN), which divides the SNN into multiple stages with progressively shrinking timesteps, and uses a temporal transformer to transform temporal scale of information. Chowdhury, Rathi & Roy (2021b) proposed an Iterative Initialization and Retraining method for SNNs (IIR-SNN) to perform single-shot inference along the temporal axis as the network gradually shrinks in the temporal domain. In particular, they also introduced spatial and temporal pruning and quantization of SNNs, with temporal pruning performed by gradually reducing simulation timesteps during training (Chowdhury, Garg & Roy, 2021a). Suetake et al. (2023) proposed a single-step SNN neuron model, demonstrating its practicality in static image classification using the surrogate gradient method.

The studies above indicate that reducing the number of timesteps effectively optimizes temporal redundancy, indicating considerable potential improvements in edge scenarios that require low latency for TSC. Based on these findings, we propose the MT-SNN architecture, which selectively masks timesteps to ensure that the network efficiently processes temporal information at various stages of training.

Network structure

Encoding methods

In the context of TSC tasks, a primary challenge is to encode complex and variable time series into spike trains using SNNs without incurring information loss. Rate encoding can approximate the original information without significant loss only in very large timesteps (Kim et al., 2018), but excessive inference time reduces energy efficiency, conflicting with the requirements for low power consumption in TSC tasks. Similarly, latency and temporal encoders face challenges related to parameter tuning or the need for additional pretraining (Chen et al., 2011). Poisson encoding introduces stochasticity by converting input intensities into spike probabilities, which will reduce reproducibility and incur additional computational cost (Rathi & Roy, 2021). Significantly, these methods typically do not achieve high performance. Due to its simplicity, direct encoding has been widely used in tasks such as image recognition (Kim et al., 2022), maintaining the integrity of the original sequence without extensive preprocessing. As one of the most effective methods for classification tasks, direct encoding can even achieve a performance comparable to ANNs (Wu et al., 2019; Zheng et al., 2021). Therefore, in the MT-SNN architecture, direct encoding is used, as illustrated in Fig. 2.

Structure of Spiking ResNet

To maximize the combination of the advantages of SNN and residual structures, average pooling, which is more consistent with the principles of SNNs, was selected as the pooling method. The classifier structure involves multiple layers including adaptive average pooling, flatten, dropout, and linear layers with spiking layers to implement the spiking mechanism. The general structure of the basic Spiking Resnet is shown in Fig. 3.

Masked timesteps

Although Spiking ResNet, obtained by directly replacing the ReLU activation function, is capable of addressing certain challenges in univariate TSC tasks compared to traditional DL and ML methods, it still encounters temporal redundancy issues. Since spiking neurons are highly correlated with the added time dimension T, in the context of direct encoding, T also represents the number of times the original data are repeated along the time axis. A larger T allows for more refined training but also leads to an increased network depth, resulting in additional computational costs. However, given the wide range of TSC data, selecting an appropriate T for each dataset is not a straightforward task. This is also why we use direct encoding to expand the time dimension rather than directly using the original data length as the time dimension. Inspired by studies on temporal redundancy that indicate that only a subset of the timesteps is significant during the training process of SNNs, we apply a masking technique to the timesteps in the Spiking Residual Blocks, thereby retaining only those timesteps that are important.

Temporal selection

In order to reduce temporal redundancy, we apply the mask M to selectively disable certain unimportant timesteps. The criterion for measuring their contribution is the firing frequency FR. The temporal reduction between Spiking Residual Blocks can be seen in Fig. 4. According to the Hebbian learning rule, a greater number of spikes indicates more substantial learning, as the frequency of these spikes characterizes the intensity of the learning process.

To calculate FR, we concatenate outputs along the time dimension to obtain the concat output $O={\oplus }_{t=1}^T{\tilde{o}}_t$.

Due to the binary nature of spiking neurons, the firing rate (FR) at timestep $t$ is obtained by summing $S_t$ across the channel and feature dimensions. This can be expressed as follows:

(2) $$FR=\sum _{c,l}{O}_{t,c,l},t=1,\dots ,T.$$

In this way, $\mathrm{F}{\mathrm{R}}_t$ is a scalar per timestep and per sample; for a batch of size B, $\mathrm{F}\mathrm{R}$ forms a matrix of size $B\times T$. The mask M is then applied to the output of each timestep:

(3) $${\tilde{O}}_t=O_t\odot M,t=1,\dots ,T$$with broadcasting along the $(B,C,L)$ dimensions, where $\odot$ denotes element-wise multiplication.

Afterward, the network retains the timesteps with high firing rates based on the predefined number of timesteps to be kept. The output, ${\tilde{O}}_t$, is then fed into the next Spiking Residual Block.

The general details of applying the masked timestep method are depicted in Algorithm 1.

Algorithm 1 :

Masked timestep processing.

Parameters:

Bi: The i-th Spiking Residual Block, $i=1,2,...,N$

N: Total number of Spiking Residual Blocks

Ti: Number of current timesteps processed by Bi

Tn: Number of next timesteps processed by $B_{i+1}$

M: Binary mask indicating active timesteps $M\in \{0,1{\}}^{T\times B\times C\times L}$

FR: Firing rates of the temporal dimension, defined as the average number of spikes per unit time

Input:

x : Input time series

Output:

$\hat{y}$: Final classification output

1: Reshape input x to $X\in {\mathbb{R}}^{T\times B\times C\times L}$ in order to fit direct encoding

2: Initialize an all True mask M with the same shape as X

3: Set initial Ti as the number of timesteps for the first block

4: for Bi; $i\le N;i++$ do

5:   Get next number of timesteps Tn for $B_{i+1}$

6:   if Tn is None then

7:     Process X without temporal masking

8:   else

9:     Clone current mask M to create $M_{current}$

10:     for $t=1,2,...,T_i$ do

11:        Compute block output Ot using the Spiking Residual Block Bi

12:        Apply mask $M_{current}$ to Ot

13:     end for

14:     Compute firing rates FR based on the block outputs

15:     Sort timesteps in Ti according to FR

16:     Select top Tn timesteps: $idx\mathrm{\_}top=idx\mathrm{\_}sorted[0\dots T_n-1]$

17:     Create new mask M such that $M[idx\mathrm{\_}top,:]=1$, others = 0

18:   end if

19:   Pass masked output through avgpooling layer

20:   Update Ti with Tn for the next block

21: end for

22: Pass the final output through the classifier layers

23: Average the output yt across the time dimension to get the final classification output $\hat{y}=\frac{1}{T_i}{\sum }_{t=1}^{T_i}{y}_t$

24: Return $\hat{y}$

DOI: 10.7717/peerj-cs.3460/table-101

TAIF neuron

In this section, the proposed Temporal Adaptive Integrate-and-Fire (TAIF) neuron is detailed.

Training strategy

One major challenge in training SNNs is the non-differentiability of spikes. In this article, we choose to use the surrogate gradient method to directly train SNNs with the BPTT method, using a differentiable surrogate function $g(x)$ to replace $\mathrm{\Theta }(x)$ during backpropagation (Neftci, Mostafa & Zenke, 2019; Lee, Delbruck & Pfeiffer, 2016). The surrogate gradient technique performs excellently, allowing for easy integration into modern deep learning frameworks and requiring neither a large number of timesteps nor extensive tuning.

In MT-SNN, for all spiking neurons, we approximate the derivative of $\mathrm{\Theta }(x)$ as

(9) $$g^{\mathrm{\prime }}(x)=\frac{\alpha }{2\left(1+{\left(\frac{\pi }{2}kx\right)}^2\right)}$$where $g^{\mathrm{\prime }}(x)$ is the derivative of

(10) $$g(x)=\frac{1}{\pi }\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\left(\frac{\pi }{2}kx\right)+\frac{1}{2}.$$

Here, $k$ is a hyperparameter used to control the shape of the function. To prevent vanishing or exploding gradients, we use $k=2$, which ensures that the maximum value of the gradient of the surrogate function is 1. The specific shape of the surrogate function is illustrated in Fig. 5.

In SNNs, commonly used spiking neurons include IF neurons and Leaky Integrate-and-Fire (LIF) neurons (Abbott, 1999). However, in TSC tasks, the leakage characteristics of LIF neurons often lead to suboptimal performance compared to IF neurons, making the IF neuron more suitable. However, the reduction in time steps in MT-SNN causes standard IF neurons to lack awareness of this change, leading to redundant processing steps. To address this issue, we propose an improved IF neuron, termed the Temporal Adaptive Integrate-and-Fire (TAIF) neuron, inspired by biological neurons. We introduce a fatigue mechanism into IF neurons, making them more prone to fatigue in the early training stages with long timesteps, while more excitable in the later stages of short timesteps. This behavior aligns with the TIC phenomenon and optimizes the temporal information distribution, enhancing their robustness and performance.

The charge function of the TAIF neuron is the same with standard IF neuron:

(11) $$\frac{d{V}_t}{dt}=I_t.$$

In simulation, we use discrete difference equations to approximate continuous differential equations:

(12) $$H_t=V_{t-1}+X_t.$$

To ensure the simplicity of IF neurons, the improvements in TAIF focus on the fire and reset behaviors. First, the TAIF charges according to the charging Eq. (12). Then, it determines whether $H_t$ has reached the threshold potential $V_{th}$. If $H_t$ exceeds $V_{th}$, a spike fired and the fatigue value $F_t$ increases by $\gamma$ and without recovery behavior, where $\gamma$ denotes the fatigue growth factor. If $H_t$ does not reach $V_{th}$, the neuron does not fire spike, and the fatigue value $F_t$ decreases by $\delta$ to simulate fatigue recovery behavior, where $\delta$ denotes the fatigue recovery rate. The accumulated fatigue value then directly affects the threshold potential $V_{th}$. The current timestep T is used as a coefficient for $F_t$, allowing the model to capture temporal changes. The update process of TAIF can be seen in Algorithm 2.

Algorithm 2 :

TAIF neuron update logic.

Input: Output from convolutional and batch normalization layers at timestep t, $X_{cb}^t$

Output: Spike train $S_t$

1: Initialize ${\gamma }_t$ and ${\delta }_t=0$

2: for $t=1$ to T do

3:   Neuron Charge:

4:   Update membrane potential Ht based on input $X_{cb}^t$

5:   if $H_t\ge V_{th}$ then

6:     Neuron Fire:

7:      $f_t={\gamma }_t$

8:     Fatigue factor update: $F_t=F_{t-1}+f_t$

9:     Neuron Reset:

10:      $V_t=H_t-V_{t{h}_t}\cdot S_t$

11:   else

12:     No spike fired

13:      $f_t=-{\delta }_t$

14:     Decrease fatigue factor Ft and ensure it does not fall below 0: $F_t=max(F_{t-1}+f_t,0)$

15:   end if

16:   Update threshold potential: $V_{t{h}_t}=V_{t{h}_{t-1}}+F_t\cdot t$

17: end for

18: Return output spike train $S_t$

DOI: 10.7717/peerj-cs.3460/table-102

In this way, we can directly train the spiking neuron layers using the BPTT method.

First, we define the FPTT function as follows:

(13) $$S_{1,...,T},V_{1,...,T}={\mathcal{F}}_{fp}(X_{1,...,T},V_0,F_0).$$

So the backward function is:

(14) $$\frac{\mathrm{d}L}{\mathrm{d}{X}_{1,\dots ,T}},\frac{\mathrm{d}L}{\mathrm{d}{V}_0},\frac{\mathrm{d}L}{\mathrm{d}{F}_0}={\mathcal{F}}_{bp}\left(\frac{\mathrm{\partial }L}{\mathrm{\partial }{S}_{1,\dots ,T}},\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_{1,\dots ,T}},\frac{\mathrm{\partial }L}{\mathrm{\partial }{F}_{1,\dots ,T}}\right).$$

Based on the TAIF update process, the computation graph is shown in Fig. 6.

The gradient of TAIF is as follows:

(15) $$\frac{\mathrm{d}L}{\mathrm{d}{H}_t}=\frac{\mathrm{\partial }L}{\mathrm{\partial }{S}_t}\cdot \frac{\mathrm{d}{S}_t}{\mathrm{d}{H}_t}+\left(\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_t}+\frac{\mathrm{d}L}{\mathrm{d}{H}_{t+1}}\right)\cdot \frac{\mathrm{d}{V}_t}{\mathrm{d}{H}_t}$$

(16) $$\frac{\mathrm{d}L}{\mathrm{d}{X}_t}=\frac{\mathrm{d}L}{\mathrm{d}{H}_t}\cdot \frac{\mathrm{d}{H}_t}{\mathrm{d}{X}_t}=\frac{\mathrm{d}L}{\mathrm{d}{H}_t}.$$

While $t$ = 0:

(17) $$\frac{\mathrm{d}L}{\mathrm{d}{V}_0}=\frac{\mathrm{d}L}{\mathrm{d}{H}_1}\cdot \frac{\mathrm{d}{H}_1}{\mathrm{d}{V}_0}=\frac{\mathrm{d}L}{\mathrm{d}{H}_1}.$$

The gradient of the loss function L to the threshold voltage $V_{t{h}_t}$ at time $t$ is:

(18) $$\frac{\mathrm{d}L}{\mathrm{d}{V}_{t{h}_t}}=\frac{\mathrm{\partial }L}{\mathrm{\partial }{S}_t}\cdot \frac{\mathrm{\partial }{S}_t}{\mathrm{\partial }{V}_{t{h}_t}}+\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_t}\cdot \frac{\mathrm{\partial }{V}_t}{\mathrm{\partial }{V}_{t{h}_t}}+\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_{t{h}_{t+1}}}\cdot \frac{\mathrm{d}{V}_{t{h}_{t+1}}}{\mathrm{d}{V}_{t{h}_t}}$$where:

(19) $$\{\begin{array}{rl}\frac{\mathrm{\partial }{S}_t}{\mathrm{\partial }{V}_{t{h}_t}}& ={\mathrm{\Theta }}^{\mathrm{\prime }}(H_t-V_{t{h}_t})(-1)\\ \frac{\mathrm{\partial }{V}_t}{\mathrm{\partial }{V}_{t{h}_t}}& =-S_t\\ \frac{\mathrm{d}{V}_{t{h}_{t+1}}}{\mathrm{d}{V}_{t{h}_t}}& =1\end{array}.$$

Therefore, the complete formula is:

(20) $$\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_{t{h}_t}}=-\frac{\mathrm{\partial }L}{\mathrm{\partial }{S}_t}\cdot {\mathrm{\Theta }}^{\mathrm{\prime }}(H_t-V_{t{h}_t})-\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_t}\cdot S_t+\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_{t{h}_{t+1}}}.$$

So we can calculate the gradients for $F_t$:

(21) $$\frac{\mathrm{\partial }L}{\mathrm{\partial }{F}_t}=\frac{\mathrm{\partial }L}{\mathrm{\partial }{F}_{t+1}}\cdot \frac{\mathrm{d}{F}_{t+1}}{\mathrm{d}{F}_t}+\frac{\mathrm{\partial }L}{\mathrm{\partial }{V}_{t{h}_t}}\cdot \frac{\mathrm{d}{V}_{t{h}_t}}{\mathrm{d}{F}_t}.$$

Parameter settings

The CSNN architecture consists of three stacked Spiking Residual Blocks followed by a classifier. Each SRB comprises two convolutional layers, batch normalization, and TAIF neuron layers. The output of each SRB is passed through an average pooling layer for downsampling. The detailed settings of the network architecture are in Table 1.

In all our tests, we used the same network structure to demonstrate the versatility of MT-SNN, and employed the same seed to ensure reproducibility. For optimization, we used Adam Optimizer with CyclicLR Scheduler (Kingma & Ba, 2017). The learning rate for Adam was set to 0.001. To encourage correct firing of neurons while silencing the others, we use MSE as the loss function.

To validate the generalizability of the MT-SNN, we adjusted only the batch size for different datasets, making it the sole hyperparameter that required adjustment, an advantage not achieved in previous works. In practical applications, one only needs to set the desired timestep reduction number. To further demonstrate the effectiveness of the proposed method, we used two timestep reduction settings (maximum timesteps of 5 or 3) in our evaluation. The evaluation results indicate that the MT-SNN achieves remarkable performance even under ultra-low timesteps (3-2-1). Detailed settings can be found in Table 2.

Our experiments were conducted using an RTX 4090 GPU and an Intel Core I9-14900K processor, operating under Ubuntu 22.04 LTS. The experimental framework used the SpikingJelly library (version 0.0.0.0.15) (Fang et al., 2023) and PyTorch (version 2.0.1) (Paszke et al., 2019). The version of the graphics card driver is 535.86.05, with CUDA 12.1.

Ablation study

In this section, detailed ablation experiments are conducted to demonstrate the effectiveness of the proposed modules. Here, MT-SNN-TAIF ( $T_3$- $T_2$- $T_1$) represents the network using Masked Timestep and TAIF, with timesteps reduced from $T_3$ to $T_2$ to $T_1$. MT-SNN-IF ( $T_3$- $T_2$- $T_1$) is the counterpart using IF neurons. SNN (T) denotes the standard SNN network without Masked Timestep and TAIF, with a fixed timestep T. The models mentioned above use the direct encoding method.

Three representative datasets were selected: Adiac, FordB, and DiatomSizeR, with $T=1$, $T=3$, $T=5$, and $T=20$ to represent timesteps ranging from very short to very long. The results are shown in Table 3. MT-SNN was not tested at $T=1$ and $T=20$ because there are insufficient timesteps to apply reduction with only one timestep. When $T=20$, selecting appropriate timesteps becomes both complex and unnecessary, as most timesteps are redundant. Additionally, MT-SNN is specifically designed for low timestep settings, making tests at high timesteps misaligned with the intended design focus.

From Table 3, it can be observed that if an appropriate timestep is selected, direct-encoding SNNs can be highly competitive, demonstrating the simplicity and efficiency of direct encoding compared to rate encoding or latency encoding in classification tasks. This observation is consistent with the results in image classification tasks. However, the performance of direct encoding method is highly related to the timesteps. In the shortest timestep, the model exhibits the lowest accuracy. As the timestep increases, the accuracy improves, but there is a point of diminishing returns where excessively long timesteps can actually degrade performance and bring extra computation cost. This indicates that the baseline model, SNN, has considerable temporal redundancy and that selecting an appropriate timestep is crucial for optimal performance. Our proposed Masked Timestep method requires only an appropriate number of timesteps. As shown in the table, the timesteps for MT-SNN, demonstrating that only five maximum timesteps can surpass the performance of SNN at 20 timesteps. However, the use of the MT method is limited by IF neurons. In the DiatomSizeR dataset, the standalone MT method even leads to a performance decline. By introducing TAIF neurons, MT-SNN consistently outperforms SNN with longer timesteps, even at ultra-low timesteps. Figure 7 indicates that in the first 100 epochs, all models except SNN (T = 20) have converged, showing that direct encoding requires more time to converge in large timesteps and exhibits noticeable oscillations in the early stages of training. As the training progresses, MT-SNN-TAIF shows a smoother and higher test accuracy compared to its counterparts at T = 3 and T = 5. This indicates that the MT-SNN-TAIF architecture has a stronger learning capacity and benefits more from additional training epochs. These results strongly demonstrate the synergistic effect of the MT method with TAIF and verify its superiority in low timestep configurations.

Comparison with previous work

In this section, we evaluate our proposed method on a total of 49 UCR univariate time series datasets. Regarding the evaluation metrics, we adopt the Mean Per-Class Error (MPCE) proposed by Wang, Yan & Oates (2017) to evaluate model classification performance in multiple datasets. For a model $M=\{m_i\}$, dataset pool $D=\{d_k\}$, class labels $C=\{c_k\}$, and error rates $E=\{e_k\}$, we have:

(22) $$PC{E}_k=\frac{e_k}{c_k}$$

(23) $$MPC{E}_i=\frac{1}{K}\sum PC{E}_k.$$

Here, $k$ indexes datasets, and $i$ indexes models. MPCE reflects the expected error rate for a single class across all datasets. Considering the number of classes, MPCE is a robust baseline criterion. We also present the average arithmetic ranking, the average geometric ranking and the critical difference diagram between MT-SNN and other methods.

Comparison with non-SNN work

• • Comparison with traditional methods:

• For traditional methods, we selected seven representative methods. DTW refers to 1NN-DTW, which is a simple and effective distance-based method but has a high computational cost and is sensitive to noise. BOSS is a dictionary-based method that handles symbolic representations well but relies on parameter settings. COTE and BOSSVS are ensemble models that combine multiple classifiers while consuming many computational resources. PROP combines 11 classifiers based on various elastic distance measures. SE1 is also an ensemble model based on discriminative shapelets. TSBF is a feature-based method that captures local patterns effectively but is influenced by segmentation and feature extraction strategies. The overall evaluation results and the CD-diagram can be seen in Table 4 and Fig. 8, respectively. Compared to traditional methods, the MT-SNN, an end-to-end, from scratch trained approach that does not require extra feature engineering, achieved the best performance. Additionally, MT-SNN does not require extensive hyperparameter tuning and exhibts tolerance to noise and the ability to learn under constrained conditions.

• • Comparison with DL-based methods: In DL-based methods, we adopt three baselines proposed by Wang, Yan & Oates (2017): the MLP, FCN, ResNet, and MCNN proposed by Cui, Chen & Chen (2016). The MLP consists of three fully connected layers with 500 neurons each, utilizing ReLU activation functions and dropout; the softmax function is employed for final classification. The FCN comprises three convolutional layers with 128, 256, and 128 filters, respectively. Each layer is followed by batch normalization (BN) and ReLU activation functions, and the global pooling layer aggregates the feature maps into a single vector. The output layer employs a softmax activation function for classification. The ResNet uses three stacked residual blocks with filters of 64, 128, and 128. The classifier consists of a global average pooling layer followed by a softmax function. The MCNN method uses downsampling, skip sampling, and sliding windows to preprocess the data for multi-scale settings, leading to significant preprocessing work and data augmentation. The overall evaluation results and the CD-diagram are presented in Table 5 and Fig. 9, respectively. Among the four models, the FCN achieves the best performance. The ResNet performs slightly worse due to its tendency to overfit. However, the MT-SNN achieves the best results with a structure similar to the ResNet, indicating that the masked timestep spiking residual block has a better generalization capacity, more reasonable biological plausibility and does not require extensive preprocessing and data augmentation used in MCNN.

Table 4:

Error rates of different traditional time series classification methods with wins, AVG arithmetic ranking, AVG geometric ranking, and MPCE.

Bold values indicate the best performance for each metric.

Err rate	DTW	COTE	BOSSVS	PROP	BOSS	SE1	TSBF	MT-SNN (Ours)
Adiac	0.396	0.233	0.302	0.353	0.220	0.373	0.245	0.117
Beef	0.367	0.133	0.267	0.367	0.200	0.133	0.287	0.125
CBF	0.003	0.001	0.001	0.002	0	0.010	0.009	0
ChlorineCon	0.352	0.314	0.345	0.360	0.340	0.312	0.336	0.196
CinCECGTorso	0.349	0.064	0.130	0.062	0.125	0.021	0.262	0.040
Coffee	0	0	0.036	0	0	0	0.004	0
CricketX	0.246	0.154	0.346	0.203	0.259	0.297	0.278	0.250
CricketY	0.256	0.167	0.328	0.156	0.208	0.326	0.259	0.185
CricketZ	0.246	0.128	0.313	0.156	0.246	0.277	0.263	0.177
DiatomSizeR	0.033	0.082	0.036	0.059	0.046	0.069	0.126	0.013
ECGFiveDays	0.232	0	0	0.178	0	0.055	0.183	0.065
FaceAll	0.192	0.105	0.241	0.152	0.210	0.247	0.234	0.174
FaceFour	0.170	0.091	0.034	0.091	0	0.034	0.051	0
FacesUCR	0.095	0.057	0.103	0.063	0.042	0.079	0.090	0.066
FiftyWords	0.310	0.191	0.367	0.18	0.301	0.288	0.209	0.184
Fish	0.177	0.029	0.017	0.034	0.011	0.057	0.080	0.038
GunPoint	0.093	0.007	0	0.007	0	0.060	0.011	0
Haptics	0.623	0.488	0.584	0.584	0.536	0.607	0.488	0.500
InlineSkate	0.616	0.551	0.573	0.567	0.511	0.653	0.603	0.720
ItalyPower	0.050	0.036	0.086	0.039	0.053	0.053	0.096	0.026
Lightning2	0.131	0.164	0.262	0.115	0.148	0.098	0.257	0.094
Lightning7	0.274	0.247	0.288	0.233	0.342	0.274	0.262	0.094
MALLAT	0.066	0.036	0.064	0.050	0.058	0.092	0.037	0.042
MedicalImages	0.263	0.258	0.474	0.245	0.288	0.305	0.269	0.228
MoteStrain	0.165	0.085	0.115	0.114	0.073	0.113	0.135	0.073
NonInvThorax1	0.210	0.093	0.169	0.178	0.161	0.174	0.138	0.098
NonInvThorax2	0.135	0.073	0.118	0.112	0.101	0.118	0.130	0.073
OliveOil	0.167	0.100	0.133	0.133	0.100	0.133	0.090	0.033
OSULeaf	0.409	0.145	0.074	0.194	0.012	0.273	0.329	0.102
SonyAIBO	0.275	0.146	0.265	0.293	0.321	0.238	0.175	0.018
SonyAIBOII	0.169	0.076	0.188	0.124	0.098	0.066	0.196	0.083
StarLightCurves	0.093	0.031	0.096	0.079	0.021	0.093	0.022	0.022
SwedishLeaf	0.208	0.046	0.141	0.085	0.072	0.120	0.075	0.082
Symbols	0.050	0.046	0.029	0.049	0.032	0.083	0.034	0.033
SyntheticControl	0.007	0	0.040	0.010	0.030	0.033	0.008	0
Trace	0	0.010	0	0.010	0	0.050	0.020	0
TwoLeadECG	0	0.015	0.015	0	0.004	0.029	0.001	0
TwoPatterns	0.096	0	0.001	0.067	0.016	0.048	0.046	0
UWaveX	0.272	0.196	0.270	0.199	0.241	0.248	0.164	0.196
UWaveY	0.366	0.267	0.364	0.283	0.313	0.322	0.249	0.272
UWaveZ	0.342	0.265	0.336	0.290	0.312	0.346	0.217	0.232
Wafer	0.020	0.001	0.001	0.003	0.001	0.002	0.004	0
WordSynonyms	0.351	0.266	0.439	0.226	0.345	0.357	0.302	0.315
Yoga	0.164	0.113	0.169	0.121	0.081	0.159	0.149	0.145
Win	3	12	4	5	13	3	4	21
AVG arithmetic ranking	6.14	2.68	5.52	4.18	3.50	5.59	4.98	2.23
AVG geometric ranking	5.48	2.25	4.76	3.63	2.80	4.98	4.34	1.83
MPCE	0.0397	0.0226	0.0330	0.0304	0.0256	0.0302	0.0335	0.0188

DOI: 10.7717/peerj-cs.3460/table-4

Table 5:

Error rates of different deep learning time series classification methods with wins, AVG arithmetic ranking, AVG geometric ranking, and MPCE.

Bold values indicate the best performance for each metric.

Err rate	MCNN (Cui, Chen & Chen, 2016)	MLP (Wang, Yan & Oates, 2017)	FCN (Wang, Yan & Oates, 2017)	ResNet (Wang, Yan & Oates, 2017)	MT-SNN (Ours)
Adiac	0.231	0.248	0.143	0.174	0.117
Beef	0.367	0.167	0.25	0.233	0.125
CBF	0.002	0.14	0	0.006	0
ChlorineCon	0.203	0.128	0.157	0.172	0.196
CinCECGTorso	0.058	0.158	0.187	0.229	0.040
Coffee	0.036	0	0	0	0
CricketX	0.182	0.431	0.185	0.179	0.250
CricketY	0.154	0.405	0.208	0.195	0.185
CricketZ	0.142	0.408	0.187	0.187	0.177
DiatomSizeR	0.023	0.036	0.07	0.069	0.013
ECGFiveDays	0	0.03	0.015	0.045	0.065
FaceAll	0.235	0.115	0.071	0.166	0.174
FaceFour	0	0.17	0.068	0.068	0
FacesUCR	0.063	0.185	0.052	0.042	0.066
FiftyWords	0.19	0.288	0.321	0.273	0.184
Fish	0.051	0.126	0.029	0.011	0.038
GunPoint	0	0.067	0	0.007	0
Haptics	0.53	0.539	0.449	0.495	0.50
InlineSkate	0.618	0.649	0.589	0.635	0.72
ItalyPower	0.03	0.034	0.03	0.04	0.026
Lightning2	0.164	0.279	0.197	0.246	0.094
Lightning7	0.219	0.356	0.137	0.164	0.094
MALLAT	0.057	0.064	0.02	0.021	0.042
MedicalImages	0.26	0.271	0.208	0.228	0.228
MoteStrain	0.079	0.131	0.05	0.105	0.073
NonInvThorax1	0.064	0.058	0.039	0.052	0.098
NonInvThorax2	0.06	0.057	0.045	0.049	0.073
OliveOil	0.133	0.60	0.167	0.133	0.033
OSULeaf	0.271	0.43	0.012	0.021	0.102
SonyAIBO	0.23	0.273	0.032	0.015	0.018
SonyAIBOII	0.07	0.161	0.038	0.038	0.083
StarLightCurves	0.023	0.043	0.033	0.029	0.022
SwedishLeaf	0.066	0.107	0.034	0.042	0.082
Symbols	0.049	0.147	0.038	0.128	0.033
SyntheticControl	0.003	0.05	0.01	0	0
Trace	0	0.18	0	0	0
TwoLeadECG	0.001	0.147	0	0	0
TwoPatterns	0.002	0.114	0.103	0	0
UWaveX	0.18	0.232	0.246	0.213	0.196
UWaveY	0.268	0.297	0.275	0.332	0.272
UWaveZ	0.232	0.295	0.271	0.245	0.232
Wafer	0.002	0.004	0.003	0.003	0
WordSynonyms	0.276	0.406	0.42	0.368	0.315
Yoga	0.112	0.145	0.155	0.142	0.145
Win	11	2	16	10	21
AVG arithmetic ranking	2.70	4.27	2.50	2.64	2.18
AVG geometric ranking	2.34	4.03	2.09	2.33	1.80
MPCE	0.0241	0.0407	0.0219	0.0231	0.0188

DOI: 10.7717/peerj-cs.3460/table-5

Visualization of spike encoder

To visualize the original one-dimensional data, the Lighting2 dataset is selected. The spike sequences of the feature maps extracted by the encoder in five time steps are plotted. For ease of visualization, the encoding of the first five channels is displayed, illustrating encoding effects at different timesteps as shown in Figs. 10, 11, 12.

Observations illustrate that, in the early timesteps, the features extracted by the spike encoder are relatively sparse. However, as the timesteps progress, more features are activated, indicating that the TAIF encoder is capable of gradually extracting and accumulating key information from the input signals over time, forming an efficient spike encoding representation. This approach avoids the non-firing issue of the LIF encoder and the misfiring issue of the IF encoder.

In addition, Fig. 13 illustrates the comparison between IF and TAIF encoders in a single channel. Through a visual analysis of feature mappings across different timesteps and neurons, the effectiveness of the TAIF spike encoder in encoding time series data becomes evident. The overlay results demonstrate the overall encoding performance, while the feature mappings of individual channels reveal fine-grained encoding characteristics. This section includes the raw data, feature mappings at various timesteps, feature mappings of individual neurons, providing a comprehensive evaluation of the encoding performance of different spike encoders.

Noise tolerance in masked timesteps

In this section, we conduct an experiment to evaluate robustness under varying levels of Gaussian noise. By introducing Gaussian noise with different standard deviations to the original time series data, we aim to analyze the impact on the performance of IF and TAIF under Masked Timestep.

The datasets used for this analysis include the CBF, Coffee and Gunpoint datasets, each subjected to three different levels of Gaussian noise: 0.5, 1.0, and 1.5.

The performance metrics for each model under different noise conditions are summarized in Fig. 14.

In the MT-SNN architecture, both models exhibit varying degrees of robustness to different levels of Gaussian noise. In the CBF and Coffee datasets, both models have a certain robustness, maintaining high accuracy across all noise levels. However, in the Gunpoint dataset, the TAIF model consistently outperforms the IF model across all noise levels. This dataset contains short sequences with localized temporal patterns, making it highly sensitive to noise. By introducing temporal adaptivity, TAIF achieves a more stable response to critical timesteps and thus preserves discriminative features more effectively, leading to greater robustness under Gaussian noise.

Training with restricted data subset

In this section, to simulate edge scenarios with limited device or training resources, real-time requirements, and costly annotation, the performance of the MT-SNN is evaluated on a restricted data subset. Unsolved binary classification tasks, which better reflect real-world situations, are selected. Their training sets are reduced to 10% of their original size, ensuring that each class has at least one sample, while the test sets remain unchanged.

To provide a thorough comparison, we also present the performance of Spiking Reservoir method using Gaussian Temporal Encoding (GTE) as described in Dey et al. (2022). Importantly, this method is trained on 20% of the training data. This comparison allows us to assess the relative performance and robustness of the MT-SNN under harsher constraints. The detailed settings and performance comparisons are provided in Tables 7 and 8.

The adaptability of the MT-SNN using direct encoding enables it to perform significantly better than the GTE using temporal encoding, even with only half the amount of training data. Despite the reduced size of the training set, the MT-SNN reaches reasonably high accuracy on several datasets with limited training samples. For datasets with a large number of original training samples, such as Earthquake, FordA and FordB, the performance of MT-SNN in a restricted setting is almost comparable to that of the full dataset, outperforming GTE under more challenging conditions. Additionally, for datasets with only a few training samples, MT-SNN still achieves acceptable performance, even reaching levels comparable to those with the full dataset in cases like MoteStrain and Lightning2. The lower results in Yoga and SonyAIBO are probably due to the inadequate representation in the 10% training data for the spike encoder. However, considering the reduction in sample size, this decrease is acceptable. In conclusion, the enhanced performance of MT-SNN under more constrained conditions highlights its robustness and adaptability compared to the Spiking Reservoir method.