Stylometry for real-world expert coders: a zero-shot approach

Model design

Due to their superior accuracy performance in stylometric tasks (Alsulami et al., 2017; Bogomolov et al., 2021), AST-based models have been widely adopted. In our study, we have chosen the code2seq (Alon, Levy & Yahav, 2018) snippet embedding structure as a common base for developing three different designs differentiated primarily by means of the attention technique, learning objective and the technique used to handle the vocabulary. We initially discuss the common part (Fig. 4) of the three architectures, an advancement of the Alon et al. (2019) model that encodes snippets by extracting stream tokens (leaves) and their ancestors from the Abstract Syntax Tree.

The pre-processing needed to extract the AST from the input code snippets is conducted using the Tree Sitter library (TreeSitter, 2022): This provides a parser generator tool capable of retrieving a representation of the snippet’s Concrete Syntax Tree which will be then simplified (e.g., by pruning redundant syntactical elements as brackets or punctuation nodes) to an Abstract Syntax Tree. In such an AST each node corresponds to one or more nodes in the concrete syntax tree as follows: the leaves correspond to the “stream tokens” (i.e., terminal nodes) while their ancestors are the non-terminal nodes. Following Alon et al. (2019) we define the AST as follows:

Definition 1 (Abstract Syntax Tree (AST)) An Abstract Syntax Tree (AST) for a code snippet C is a tuple 〈N, T, X, s, δ, ϕ〉 where: N is a set of non-terminal nodes, T is a set of terminal nodes, X is a set of values, s ∈ N is the root node, δ:N → (N∪T)^∗ is a function that maps a non-terminal node to a list of its children, ϕ:T → X is a function that maps a terminal node to an associated value. Every node, except the AST root, appears exactly once in all the lists of children.

The architecture uses, as a primary source of input, triplets of elements extracted from the AST, composed by random pairs of terminal nodes which will be split into subtokens, and their channeling path, an ordered list of the terminal nodes’ ancestors (non-terminal nodes) extracted by ascending the AST until finding the lowest common ancestor.

To split the terminal nodes into subtokens, we introduce a mixed usage of regular expression (splitting the terminal nodes’ tokens by means of camel notation or underscores, e.g., from arraySplitter to array, Splitter), which was already adopted in the original code2seq design, and a subsequent lower-cased SentencePiece encoding (see Kudo & Richardson (2018), pruned by the space inclusive underscore character in order to maintain a stream token representation in line with Alon, Levy & Yahav (2018). In this paper, we introduce a novel use of this technique in order to try to overcome the open-vocabulary problem caused by variable names, thus gaining capabilities of unknown and rare words encoding and therefore using more effectively the code2seq subtoken architecture, which sums the subtokens embedding representation.

Having the aim to disambiguate authors by means of coding style, we decided to strip comments into a common ‘comment’ token that emphasizes only the decision of the author of commenting on the code, eliminating thus biases in the inference phase given by the potential presence of Natural language.

We now discuss how the architectures that we use in this work handle the previously defined AST representation of code snippets. Given as input a predefined set of triples L = {L₁, …, L_n}, let us consider the ith path where X_sTi ∈ ℝ^|X_sTi|×d = [t₁, …, t_m] is the starting sequence of subtokens (with an embedding dimensionality d which defines the number of parameters used), while and X_eTi ∈ ℝ^|X_eTi|×d is the ending sequence of subtokens. The subtokens are encoded with an element-wise summation in the embedded representation to obtain the following: ${\displaystyle T_{Si}={\sum }_{t\in X_{sTi}}t}$ ${\displaystyle T_{Ei}={\sum }_{t\in X_{eTi}}t}$

The AST path input, corresponding to a set of non-terminal tokens defined as X_ASTi ∈ ℝ^|X_ASTi|×d, is encoded by means of a bidirectional LSTM as follows: ${\displaystyle h_{1i},\dots ,h_{li}=LSTM\left(X_{AST1i},\dots .,X_{ASTli}\right)}$ where h_li corresponds to the lth LSTM’s hidden state representation of the ith triple.

We denote by ${\displaystyle ASTrep{r}_i=\left[h_{li}^{\to },h_{1i}^{\leftarrow }\right]}$ where ASTrepr ∈ ℝ^1×2d, the result of the concatenation of the final hidden states of the bidirectional LSTM.

To combine the triplet in a context vector, we first use a W_ast ∈ ℝ^2d×d projection matrix to reduce the dimensionality of ASTrepr to the dimension d, thus obtaining: ${\displaystyle ASTre{d}_i=ASTrep{r}_i\cdot W_{ast}.}$

The next step is then mapping the concatenated encoding of the two stream tokens and the AST representation [T_Si, ASTred, T_Ei] to the context vector. In order to achieve this we first reduce the dimension by using the (multiplication by the) W_ctx ∈ ℝ^3d×d context projection matrix and then we use the ReLU activation function: ${\displaystyle ContextVecto{r}_i=ReLU\left(\left[T_{Si},ASTred,T_{Ei}\right]\cdot W_{ctx}\right)}$

It is worth noting that differently from the original code2seq and code2vec architectures (Alon, Levy & Yahav, 2018; Alon et al., 2019), we have chosen to use the ReLU activation function rather than the tanh, maintaining the tanh function exclusively in the soft attention based classification model. This choice is motivated by the fact that using the tanh activation function we experienced a much longer time for the convergence of the model in the training procedure.

So far, we have described the part of the architecture that is common to the three models we have considered. Next, we describe the parts that differentiate the models, namely the attention mechanism, the learning objective, the dropout presence, and the use of the SentencePiece technique.

The model named as softAttn-classifier (Fig. 5) relies on the SentencePiece technique with a compressed vocabulary of 64,000 different tokens for the terminal nodes and non-terminal nodes related vocabulary of 174 different tokens (without any term of compression). The absence of the SentencePiece technique on the non-terminal nodes pre-processing phase is common to the three architectures. This is due to the strict closed-vocabulary use case, which implies the absence of out-of-vocabulary tokens (OoV). This model uses the soft-attention mechanism introduced by code2vec (Alon et al., 2019), creating thus a hybrid version between the advancements of code2seq in terms of sparseness-related issues with the code2vec-shaped output. After the attention mechanism, we have applied an M.L.P. layer with a ReLU activation function (512 →ReLU →512 →ReLU →256) gaining non-linearity advantages (Appalaraju et al., 2020) for the final vectorial representation, useful for a k-NN classification (see ‘Classification process’). We then have a final classification head with A = 104 nodes as the overall number of authors obtained from the in-distribution dataset in the former mining process. We have conducted the training process with the Cross-Entropy loss. The model has respectively an embedding (previously referred to as d) and LSTM hidden representation of 256 units. We have then exploited Dropout as a data regularization factor (0.25 right after the context embedding concatenation).

The second model, referred to as selfAttn-classifier (Fig. 6) shares the same vocabulary compression algorithm as the previously depicted model for the terminal nodes, having the SentencePiece technique with a compressed vocabulary of 64,000 tokens and a non-terminal nodes related vocabulary of 174 different tokens. This model replaces the soft attention technique with a self-attention mechanism (transformer architecture (Vaswani et al., 2017) inspired by the design used in the work of Radford et al. (2021). Since the code2seq model utilizes random pairs, no positional encoding and no autoregressive masking are added, preserving the core random representation concept of code2seq by leveraging the positional invariance properties of the architecture. The output of the transformer is subsequently normalized with a Layer normalization technique, flattened, re-projected to a lower dimensionality of 512 units using a ReLU activation function, and then passed to the classification head (with A = 104 nodes). We have conducted the training process with a Cross-Entropy loss. The model consists of an embedding (previously referred to as d) and Bi-LSTM hidden representation of 512 units, along with a transformer architecture comprising a 12-layer, 512-wide model with 8 attention heads, as described by Radford et al. (2021).

The third model, named ML model (like Fig. 5, but without the final classification head), as we experienced better performances, does not rely on the SentencePiece algorithm, having thus a vocabulary of 147,967 tokens for the terminal nodes and, like the preceding architectures, encompasses 174 different values for the non-terminal ones. This model relies on the soft-attention mechanism as Alon et al. (2019). After the attention mechanism, we have then applied an M.L.P. layer with a ReLU activation function (512 →ReLU →512 →ReLU →256) gaining non-linearity advantages (Appalaraju et al., 2020). The training process, in this case, is conducted with the infoNCE loss (further training details on ‘Training’) introduced by van den Oord, Li & Vinyals (2018). The model relies on an embedding d and LSTM hidden representation of 256 units and uses a dropout regularization of 0.25 after the context embedding concatenation.

Training

We trained the first two classification models (softAttn-classifier and selfAttn-classifier) using a stochastic gradient descent optimizer with a learning rate scheduler. We set the initial learning rate to 0.01, and the scheduler followed a step size of 50 epochs with a gamma value of 0.95. Each training batch had a dimension of 64 elements. The softAttn-classifier model incorporates input boundaries, following the approach described by Bogomolov et al. (2021). Specifically, we used AST boundaries of four units regarding the distance between leaves and a maximum path length (composed of non-terminal tokens) of seven elements. The two models were trained for 1200 epochs, and the best checkpoint, determined by the best results in terms of validation loss, was saved and used for testing purposes.

The ML model, trained with the infoNCE loss proposed by van den Oord, Li & Vinyals (2018), modified by Zhang et al. (2022) operates on N pairs of distinct snippets from the same class. We obtain batches of dimensions N², where the matrix diagonal of the batch represents the ground truth over the snippets representations on the latent space (v_i, u_i) as the i-th pair (Fig. 7). To measure the distance between embeddings, we employ the cosine similarity with the L2 norm: ${\displaystyle 〈v_i,u_i〉=\frac{v_i^T{u}_i}{\left|\right|v_i\left|\right|\left|\right|u_i\left|\right|}}$ We then apply the cosine similarity in the following loss function: ${\displaystyle {\ell }_i^{\left(v\to u\right)}=-log\frac{exp\left(〈v_i,u_i〉/\tau \right)}{{\sum }_{k=1}^{N}exp\left(〈v_i,u_k〉/\tau \right)}}$ ${\displaystyle {\ell }_i^{\left(u\to v\right)}=-log\frac{exp\left(〈u_i,v_i〉/\tau \right)}{{\sum }_{k=1}^{N}exp\left(〈u_i,v_k〉/\tau \right)}}$ ${\displaystyle Loss=\frac{1}{N}{\sum }_{i=1}^N\left({\ell }_i^{\left(v\to u\right)}+{\ell }_i^{\left(u\to v\right)}\right)/2}$

The final loss is calculated as the mean of the two previously derived losses, following the approach established by Radford et al. (2021). The parameter τ denotes the temperature parameter that governs the range of logits in the softmax function; we fixed it at 0.1.

Given that the batch determines the negative and positive samples, it is imperative to customize the sampler to ensure that only distinct classes are selected to prevent the divergence in the latent space of elements belonging to the same class. Consequently, this introduces a limitation on batch dimensionality during training, where the total number of classes present in the dataset determines the maximum batch size.

We have trained the metric learning-based model (ML model) for 1,200 epochs with Adam optimizer, an initial learning rate of 0.01, applying a scheduler with a step size of 50 epochs and a gamma value of 0.95. Each training batch had a dimension of 64² elements. As the softAttn-classifier model, we incorporated input boundaries, using AST boundaries of 4 units regarding the distance between leaves and a maximum path length of 7 elements. As the former models, we kept the best checkpoint in terms of validation loss.

Classification process

We utilized the latent space representations obtained from the code snippets for the classification task to calculate similarities using a k-NN classifier. We employed the cosine similarity metric to measure similarity, aligning with the ML model for consistency. This choice of similarity metric, as highlighted by Horiguchi, Ikami & Aizawa (2020), resulted in improved embedding representations even for the cross-entropy trained models.

To obtain the latent space representation, we removed the classification head from both the softAttn-classifier and selfAttn-classifier models, resulting in a dimensionality of 256 for the softAttn-classifier model and 512 for the selfAttn-classifier model. The ML model retained a dimensionality of 256 units. We have settled the k-NN classifier to work as a 1-NN classifier, using the closest representation as the inferred outcome. We have processed the data for the NN classifier by projecting in the latent space and computing similarities of each snippet of code from the testing set when it comes to evaluating the in-distribution properties, and zero-shot test set for the out-distribution properties; discarding then the first result as it would always be the querying snippet representation. To provide a basis for comparison with previous work by Dauber et al. (2018) and Kurtukova, Romanov & Shelupanov (2020), we have conducted the experiments with author subsets consisting of five, 10, 20, and 104 authors. To ensure randomness, a random author picker was employed. We have performed five experiment iterations, and the results were reported with the mean and standard deviation, considering the significant source of variance introduced by the random author selection.