Sa-SNN: spiking attention neural network for image classification

SECA module for channel attention

Traditional attention mechanisms, such as SENet, allocate weights between channels through fully connected layers. This process introduces a large number of parameters, especially when processing high-dimensional feature maps, which increases computational complexity. In order to solve this problem, we propose a method based on local cross-channel interaction, it refers to local interactions between different feature channels rather than capturing attention weights through global channel relationships. Typically in a convolutional neural network, each feature map (channel) processes the input independently. Cross-channel interaction means that channels are no longer completely independent, but influence each other. The Spiking-Efficient Channel Attention (SECA) mechanism calculates the relationship between channels through local interactions without introducing additional parameters or complex global pooling. The core innovation of SECA is that it introduces a parameter-free local one-dimensional convolution to realize the interaction between channels, simplifying the calculation of the attention mechanism while maintaining the ability to identify important features.

Following a thorough examination of the impacts stemming from channel dimension reduction and cross-channel interaction within the realm of channel attention learning, the conclusions drawn from these analyses lay the foundation for the introduction of the SECA module (Wang et al., 2020). The architecture of this module is visually depicted in Fig. 5, illustrating how the insights garnered from the earlier analyses are translated into a concrete design aimed at optimizing channel attention mechanisms.

When presented with an aggregated feature denoted as y ∈ ℝ^C, where C represents the channel dimension, the channel information can be propagated or carried forward through the network architecture. (8)${\displaystyle \omega =\sigma \left(\mathbf{Wy}\right).}$

In this context, W represents a parameter matrix with dimensions C × C. Within this paper, an approach is employed to capture local cross-channel interactions that emphasize efficiency and efficacy. Specifically, a band matrix denoted as W_k is utilized to learn channel attention. This band matrix W_k (9)${\displaystyle w^{1,1}\cdots w^{1,k}00\cdots \cdots 00w^{2,2}\cdots w^{2,k+1}0\cdots \cdots 0⋮⋮⋮⋮\ddots ⋮⋮⋮0\cdots 00\cdots w^{C,C-k+1}\cdots w^{C,C}}$

encompasses k × C parameters, as evident in Eq. (9). This matrix is typically relatively small in size. Moreover, Eq. (8) avoids complete isolation between distinct groups of interactions. Equation (9) considers only the interactions between y_i and its “ k” neighbors, effectively capturing local dependencies. This approach fosters a balance between preserving relevant information within the local context and maintaining computational efficiency. (10)${\displaystyle {\omega }_i=\sigma \left(\right.\sum _{j=1}^k{w}_i^j{y}_i^j\left)\right.,y_i^j\in {\Omega }_i^k.}$

Here, ${\Omega }_i^k$ signifies the collection of “ k” neighboring channels associated with the channel y_i. This set encapsulates the specific subset of nearby channels that contribute to the local cross-channel interactions involving the particular channel y_i.

A more efficient approach would be to have all channels share the same learning parameters, i.e.: (11)${\displaystyle {\omega }_i=\sigma \left(\right.\sum _{j=1}^k{w}^j{y}_i^j\left)\right.,y_i^j\in {\Omega }_i^k.}$

Note that this strategy can be easily realized by fast 1D convolution with kernel size k, i.e., (12)${\displaystyle \omega =\sigma \left({\mathrm{C1D}}_k\left(\mathbf{y}\right)\right).}$

In this context, C1D refers to a 1D convolution operation. The approach outlined in Eq. (12) is implemented through the utilization of the SECA (spiking-efficient channel attention) module, characterized by only k parameters. Within this module, the chosen value is k = 3, a decision made to ensure the optimal balance between model complexity and the capability to effectively capture local cross-channel interactions (Wang et al., 2020). The choice of k = 3 enhances efficiency while maintaining the effectiveness of the module. The PyTorch implementation is shown below:

 
 
       Listing 1: SECA Attention 
avg_pool  = AdaptiveAvgPool2d(1) 
conv = Conv1d(1, 1,  kernel_size=kernel_size, 
                 padding=(kernel_size -  1) // 2, bias=False) 
sigmoid = Sigmoid() 
def  forward(x): 
     y =  avg_pool(x) 
     y = conv(y.squeeze(-1).transpose(-1, -2)) 
     y = y.transpose(-1, -2).unsqueeze(-1) 
     y = sigmoid(y) 
     return  x *  y.expand_as(x)    

We adopt a local cross-channel interaction strategy, but only one channel is used in the experiments. The reason for choosing this approach is to simplify the analysis of cross-channel interactions, allowing us to focus on fundamental aspects of the interaction without being distracted by the complexity of multiple channels. The image classification task targeted in this article has been proven to achieve remarkable results under single-channel input.

Given our incorporation of the efficient channel attention (ECA) module (Wang et al., 2020), a meticulously designed construct adept at effectively capturing localized cross-channel interactions, the necessity to assess the scope of these interactions (indicated by the kernel size k in the context of 1D convolution) becomes imperative. Across diverse CNN architectures (O’Shea & Nash, 2015), the optimum span of interaction for cross-validation purposes can be subject to manual adjustments, aimed at accommodating convolutional blocks characterized by varying channel quantities. Nonetheless, this manual fine-tuning procedure, executed via cross-validation, has the potential to impose substantial computational demands.

It is noteworthy that the application of group convolutions has yielded significant enhancements to CNN architectures. In these configurations, channels of high dimensionality (or low dimensionality) are subjected to convolutional operations spanning expansive (or restricted) domains, contingent on a predetermined quantity of groupings. Drawing upon analogous concepts, a plausible supposition emerges: the extent of interaction, denoted by the kernel size k in 1D convolution, might demonstrate a correlation with the channel dimension C. This correlation is conceptualized through a functional relationship denoted as φ, which potentially delineates the connection between k and C: (13)${\displaystyle C=\varphi \left(k\right).}$

This functional expression φ encapsulates the dynamics by which the optimal kernel size k evolves in congruence with fluctuating channel dimensions. This construct furnishes a framework for adapting interaction ranges to distinct architectural contexts, obviating the necessity for extensive manual adaptations and resource-intensive cross-validation procedures.

The most basic form of mapping involves a linear function, expressed as φ(k) = γk − b. Nevertheless, the scope of this relationship is notably constrained within the confines of a linear function. Conversely, it is widely acknowledged that the channel dimension C—also referred to as the number of filters—is often configured as a power of two. Consequently, an alternative avenue is introduced by imbuing the linear function φ(k) = γk − b with nonlinearity. This step aims to leverage the inherent characteristics of power-of-two channel dimensions to enhance the expressiveness and adaptability of the relationship. (14)${\displaystyle C=\varphi \left(k\right)=2^{\left(\gamma \ast k-b\right)}.}$ Subsequently, when presented with the channel dimension C, the appropriate kernel size k can be dynamically ascertained through an adaptive process by: (15)${\displaystyle k=\psi \left(C\right)={\left|\frac{lo{g}_2\left(C\right)}{\gamma }+\frac{b}{\gamma }\right|}_{odd}.}$ Here, |t|_odd signifies the nearest odd number to the value of t. In the context of this research, we uniformly assign the values of γ and b as 2 and 1, respectively, across all experimental scenarios. Evidently, employing this mapping strategy results in a situation where high-dimensional channels experience interactions spanning greater distances, whereas low-dimensional channels engage in interactions spanning shorter distances. This outcome is achieved by means of the nonlinear mapping, which optimally adapts the kernel size according to the channel dimension, thereby enhancing the efficacy of cross-channel interactions.

Although ResNet’s residual connection effectively compensates for the loss function problem in deep networks, the SECA attention mechanism further improves feature extraction capabilities. By adaptively adjusting the weight of the channel, SECA emphasizes the important information in the input features, allowing the model to have better expression ability and generalization performance when facing complex data. In our series of experiments, the loss function of the model was significantly reduced after adding SECA, verifying the necessity of SECA in optimizing network performance and improving feature selection.

Spiking residual attention neural networks

The basic architecture of the SNNs model consists of a data input layer, residual blocks, LIF neurons, an attention mechanism (SECA), and a fully connected layer. The inputs to the model are the dataset of image classes and the DVS dataset, and the outputs are the classes of the target. The input layer receives the image to be categorized, and after passing through the spiking neurons, the image is encoded into a sequence of spikings with a certain time step. The pulse sequence is propagated forward between the layers, and the pulse sequence generated in the pulse output layer is accumulated over the time step and used to achieve the final classification decision. The spiking residual attention neural network designed in this work is shown in Fig. 6. The final structure of our model is composed of five residual blocks, three LIF neurons, a SECA module, a dropout layer, and a fully connected layer. The structure of the model is the best position that we have obtained through our experiments.

In this paper, we introduce LIF neurons between different ResNet blocks to enhance the transfer and integration of information. LIF neurons effectively maintain the flow of information on important features through the spiking mechanism, thereby alleviating the vanishing gradient problem in deep networks. In addition, LIF neurons are able to capture temporal dependencies through dynamic spike sequences when processing time series data, further improving the model’s performance in complex tasks. Experimental results show that the network using LIF neurons has achieved significant performance improvements in image classification, verifying its potential in deep learning.