Deep learning-based automatic action extraction from structured chemical synthesis procedures

The formal definition of the task

The data

Vectorization

Vectorization is a crucial technique that involves transforming raw data into a numerical form that can be fed into ML algorithms. Vectorization, in our case, is necessary as we use neural networks that rely on mathematical operations to process the data and make predictions. Our research objective is texts (which we have in both tasks’ input and output for Task 2). The section below describes each task’s applied vectorization process in more detail.

Vectorization for Task 1

The difficulty of vectorizing text about organic chemistry is because it contains many rare or unseen terms, such as newly discovered compounds or experimental results, molecular formulas, and chemical reactions that are not included in existing vocabularies (Perera, Dehmer & Emmert-Streib, 2020). The problem is generally called an open vocabulary problem and refers to the challenge of dealing with an infinite or unknown set of possible inputs. Open vocabulary problems can pose significant challenges for ML models, requiring them to handle previously unseen inputs that may not conform to known patterns or rules (Eichstaedt et al., 2021; Mielke et al., 2021). It also automatically excludes standard closed vocabulary methods, such as Bag-of-Words (BoW) (Qader, Ameen & Ahmed, 2019) or one-hot encoding (Zhang & LeCun, 2017). One possible approach is to use SentencePiece tokenizer, a method that breaks down the text into smaller subunits, such as individual characters or word pieces, and then represents these subunits as numerical vectors. It is also consistent with the fact that chemical nomenclature is composed of word fragments that describe specific functional groups and the sub-structure of the molecule. For this reason, the SentencePiece tokenizer has been selected as one of the most appropriate for the project (Mugisha & Paik, 2022; Abdel-Aty & Gould, 2022). Another possibility would be to use character-level embeddings when text is represented as a sequence of vectors, where each vector corresponds to a character in the text (Gajendran & Sugumaran, 2020). However, vectorizing long paragraphs can be computationally expensive and may not be the most efficient way to represent text data, so it is not considered in our research article.

Preparing a SentencePiece tokenizer starts with collecting the entire corpus of text data, preprocessing by removing any non-textual elements, such as HTML tags or special characters, and converting the text to lowercase if necessary. The sentencepiece library was used in our research article. Next, the text is segmented into variable-length subword units using unigram language modeling (Park et al., 2021). The subword units are then sorted based on their frequency of occurrence in the corpus, and a predefined number of units are selected to form the final vocabulary. Next, the training process starts, and the subword segmentation and vocabulary generation steps are repeated iteratively on the corpus until convergence to maximize the likelihood of the observed text data given the subword vocabulary. The SentencePiece algorithm has several essential parameters that can be customized for optimal performance. This research tested six vocabulary sizes, i.e., 2,500, 5,000, 10,000, 16,000, 32,000, and 64,000. The larger vocabulary can potentially improve the accuracy (because even rare subwords can be recognized and vectorized), but it also requires more computational resources. The algorithm can also split digits so that individual digits and corresponding numerical integers encode each number. However, texts with formulas and complex nomenclature often have several numbers in a single sentence. Therefore, it is advisable not to vectorize with split digits to preserve the size of the final vector. The process generates a vocabulary of set size which then was used to vectorize the input data. The output is a binary label and does not require any additional processing.

Vectorization for Task 2

For the second task, we used the same vectorization technique, i.e., the SentencePiece tokenizer, to vectorize both input and output data. This method was chosen because, similarly to the first task, the input data is in text form and contains several unique and domain-specific terms related to organic chemistry procedures, compounds, and solvent names. The output data is also text-based but structured, representing actions and parameters of actions. Therefore, a suitable vectorization method was necessary to represent the text data in a way ML methods could process. The vocabulary of input and output is almost identical; therefore, the SentencePiece model has been trained from the entire dataset. Due to it, shared embedding layers with Transformers may be used for ML methods. The SentencePiece vectorization method effectively and efficiently represents the input and output data for translating organic chemistry procedure sentences into structured sentences.

Deep learning methods

Deep learning (DL) is a type of ML that has gained significant attention in recent years due to its ability to approximate the relationships between input and output data. DL models are composed of multiple layers of interconnected units, which can learn hierarchical representations of the input data. The DL has applications in various research fields, including computer vision, natural language processing, and drug discovery. However, the effectiveness of DL methods can vary significantly depending on various factors, such as the nature of the solving task, dataset diversity, and other important characteristics. In the context of our research, we aim to classify and convert text sequences, so it is essential to carefully consider the types of methods used to train DL models. In the following section, we present the most suitable ones for our solving tasks.

Deep learning methods for Task 1

For classifying patent paragraphs into chemical procedures or other descriptions, we explored the following types of ANNs:

• GRU (gated recurrent unit) is a type of recurrent neural network introduced to address some of the limitations of standard RNNs in processing sequential data. One of the main limitations of standard RNNs is the vanishing gradient problem (Rehmer & Kroll, 2020), which can make it difficult for the network to learn long-term dependencies in the data. GRUs use gated units to control the flow of information through the network. However, GRUs have few parameters, making them more computationally efficient and easier to train. GRUs also have a relatively simple architecture, making them a good choice for modeling short-term dependencies in the data. In particular, the reset and update gates in GRUs allow the network to selectively forget or update information from the previous time step, which can help model sequential data (Gruber & Jockisch, 2020). Therefore, GRUs have been chosen as the most basic type of architecture for this task.

• LSTM (long short-term memory) is a recurrent neural network for handling sequential data with long-term dependencies. Similarly, to RNNs, LSTMs have a short-term memory connected to weights and biases, but unlike standard RNNs, LSTMs have cell states where memories flow through time steps and do not affect the gradient. The memory cell in LSTMs has three gates that allow it to selectively add, remove, or retain information at each time step. These gates are known as the input gate, forget gate, and output gate (Staudemeyer & Morris, 2019). The input gate determines how much new information is added to the memory cell at each time step, the forget gate decides what information is removed from the memory cell, and the output gate controls how much of the memory cell is used to generate the output at each time step.

• BiLSTM (bidirectional LSTM) is a variation of LSTM that processes input data in both forward (from past to future) and backward (from future to past) directions (Alawneh et al., 2020). In addition, BiLSTMs have mechanisms allowing to merge models of both input directions, making this feature particularly useful for tasks where context and dependencies from both directions are important.

• Transformer. The traditional transformer neural network architecture (Lakew, Cettolo & Federico, 2018; Shao et al., 2019) consists of the encoder and decoder blocks and relies on attention mechanisms. Self-attention (used in encoder and decoder blocks) allows the model to better “understand” how tokens in the sequence depend on each other. This is done by computing a set of attention scores between each token in the sequence and every other token, which are then used to compute a weighted sum of the input embeddings. The encoder-decoder attention is responsible for focusing on the appropriate parts in the input sequence when making the predictions (Liu et al., 2018). In addition to attention, the transformer also uses residual connections and layer normalization, which help mitigate the vanishing gradient problem and improve the stability of the training process. For this task, we trained the transformer model from scratch, which consists of an encoder connected to the feed-forward layer. Afterward, the feed-forward layer is connected to the output layer, producing a single numerical value for binary classification. Finally, each model was explored to determine the best fit for the classification task.

Deep learning methods for Task 2

For converting sentences of chemical procedures to structured and simplified formats, we explored the following types of ANNs.

• Seq2seq LSTM is a type of neural network architecture designed to learn mappings from an input sequence to an output sequence of variable length. This is achieved using two separate LSTMs: an encoder network that processes the input sequence and generates a fixed-length context vector and a decoder network that uses the context vector to generate the output sequence.

• Seq2seq BiLSTM. Similarly to seq2seq LSTM, this approach used BiLSTMs instead of LSTMs by processing the input in the encoder in both directions (Ro et al., 2022; Egonmwan & Chali, 2019). We anticipate that the bidirectional aspect of the model should better capture the context and dependencies in the input sequence, which can lead to improved performance. The decoder layer contained LSTM.

• Transformer. We have used the traditional transformer architecture adjusted for our seq2seq problem (Chi et al., 2021; Jawahar et al., 2021), allowing input and output sequences of different lengths, such as used in, e.g., machine translation (Garg et al., 2021; Senadeera & Ive, 2022). In this architecture, the encoder takes the input sequence and generates a series of context vectors, one for each time step in the input sequence. The decoder then takes these context vectors as input and generates the output sequence one token at a time. At each time step, the decoder uses an attention mechanism to focus on the relevant parts of the input sequence, allowing it to generate an output token informed by the entire input sequence. One of the key advantages of this architecture is its ability to handle long-range dependencies in data, allowing it to capture complex patterns in sequences of arbitrary length. We have utilized the model described in an article by Vaswani et al. (2017) for this task. An open-source Python library OpenNMT-tf was utilized to implement the tokenization algorithms and training.

• T5 (text-to-text transfer transformer) architecture is one of the most advanced in the field of natural language processing (NLP), offering an adaptable approach to various tasks (Mars, 2022). T5 employs an encoder-decoder architecture like other seq2seq transformer models. The encoder processes input text and generates a series of context vectors, while the decoder attends to these context vectors to produce the output text (Hui et al., 2022). Incorporating self-attention mechanisms enables the model to effectively capture long-range dependencies and relationships between words in input and output sequences. An essential aspect of the T5 architecture is implementing a masked language modeling technique during the pre-training phase (Young & You, 2023; Wettig et al., 2022). This method involves masking specific input text segments, and the model predicts the omitted words. As a result, the model acquires an understanding of language structure and semantics. A pre-trained model is often used to fine-tune a particular task to bypass the computationally intense task of training the model from scratch. Its capacity to capture long-range dependencies and understand complex language patterns makes it an optimal choice for numerous NLP applications (Bird, Ekárt & Faria, 2021; Najafi & Tavan, 2022), such as text translation for our task. We have utilized the HuggingFace library for this task: base model-(https://huggingface.co/t5-base, accessed 12 December 2022), small model-(https://huggingface.co/t5-small, accessed 12 December 2022).

Hyper-parameters and optimization

After correctly defining the task (see “The formal definition of the task” section) and choosing promising types of methods (see the “Materials & Methods” section), the choices of their hyper-parameter values are no less important. Moreover, the optimal values of hyper-parameters play a crucial role in the model training process (Yu & Zhu, 2020) as they directly affect the model’s evaluation results. In this research, we have investigated the following hyper-parameters and their values:

• Activation functions (investigated values: ReLU, GELU, SELU, ReLU, ELU, and tanH) (Hendrycks & Gimpel, 2016; Rasamoelina, Adjailia & Sincak, 2020). Activation functions determine how the model will process and transform input data.

• Optimizer (Adam, Nadam, SGD, Adamax, and RMSprop) (Bischl et al., 2023). The optimizer is responsible for updating the model weights during training, and different optimizers can affect the speed and quality of the learning process.

• Batch size (16, 32, 64, 128, 256, 512). The batch size determines how many samples are processed in each training process iteration. Therefore, different batch sizes can affect the speed and quality of the learning process (Smith et al., 2017).

• Attention heads (2, 4, 6, 8) (for Transformer only) Varying the number of attention heads impacts the model’s capacity to process information efficiently, with a trade-off between performance and computational resources.

• Neural network layer size (16, 32, 64, 128, 256, 512) Layer size influences the model’s complexity and expressive power, with larger sizes potentially offering better performance at the cost of increased computational resources and overfitting risk.

Hyperparameter optimization in deep learning models involves systematically searching and selecting optimal hyperparameters. This process used a grid search technique to navigate the hyperparameter space and evaluate various configurations. The goal is to identify the hyperparameter combination that minimizes the model’s generalization error and maximizes its performance on unseen data. We have utilized a technique of early stopping to stop the training process if the validation accuracy did not increase in the seven latest epochs. All models used a loss function of binary cross-entropy in the output layer. We used the wanDB platform (www.wandb.ai, accessed on 6 September 2022), which provides visualization of multiple runs and is convenient for navigation and analysis. The following hyper-parameters were optimized:

Optimized parameters for Task 1

Neural network layer size: 16, 32, 64, 128, 256, 512.

Embedding dimensions: 4, 6, 8, 16, 32.

Activation functions: GELU, SELU, ReLU, ELU, and tanH.

Optimizers: Adam, Nadam, SGD, Adamax, and RMSprop.

Dropout: an interval of 0.1 to 0.8.

Sample importance factor: interval of 0.5 – 3.0.

Batch sizes: 32, 64, 128, 256.

Only for transformer architecture. The number of heads: 2, 4, 6, 8 and the number of layers 1, 2, 3, 4.

Optimized parameters for Task 2

Neural network layer size: 16, 32, 64, 128, and 256.

Embedding dimensions: 4, 6, 8, 16 and 32.

Activation functions: GELU, SELU, ReLU, ELU, and tanH.

Optimizers: Adam, Nadam, SGD, Adamax, and RMSprop.

Batch sizes: 32, 64, 128, 256.

Transformer architectures were not additionally optimized due to a significantly higher computational cost, and only different vocabulary sizes were tested. A pre-trained T5 model was used, and therefore hyper-parameters were not changed.

Optimal method architectures and hyper-parameters

The models were trained to employ the most suitable parameters and architectures identified through the optimization process for evaluation. This section describes the optimal parameters and respective architectures for both tasks. The GitHub repository (https://github.com/Mantas-it/ActionExtraction) provides access to the corresponding models and their optimal parameter combinations.

Summary of optimal hyper-parameters for Task 1

The GRU, LSTM, and BiLSTM neural network architectures are composed of the embedding layer, a dropout layer, a corresponding GRU/LSTM/BiLSTM layer, and a feed-forward (FF) layer connected to the output in sequence. Table 3 illustrates the optimal hyper-parameters for all combinations with different vocabulary sized. The optimal activation function of the FF layer was found to be ReLU in all cases. Similarly, a batch size of 64 led to the best performance and is not displayed in the table. The embedding dimension represents the number of dimensions the embedding layer uses. The importance value signifies how much more important are the samples that belong to the class of real procedures because the training dataset is not balanced.

The transformer neural networks for Task 1 consist of the embedding layer and one or more blocks of multi-head attention layer combined with the normalization layer. Table 4 presents relevant hyper-parameters for various vocabulary sizes. The dropout was set to zero because higher values led to an unstable training process. The attention layers number describes how many multi-head attention layers were used.

Summary of optimal hyper-parameters for Task 2

The seq2seq LSTM and BiLSTM architectures are constructed by connection encoder and decoder, which themselves are made up of an input layer, an embedding layer, and a corresponding LSTM or BiLSTM layer. Table 5 describes the hyper parameters that were found to be optimal for different vocabulary sizes. A batch size of 64 was used for the training process to conserve memory use.

The transformer model for the translation task was implemented via the OpenNMT-tf library (Materials & Methods). Although the models were only trained with a vocabulary of different sizes, the model architecture was not optimized. The model consists of an encoder and a decoder, each with six stacks. Each stack consists of a multi-head attention layer (eight heads and a size of 512), a normalization layer, and a feed-forward layer (of size 2,048). Other values were left as default, and only the learning rate (0.005), batch size (4,096), and effective batch size (32,768) were modified. The T5 small and base models (Materials & Methods) were only fine-tuned and were not subjected to hyperparameter optimization.

Results of Task 1

As discussed in the Methods & Materials section, the binary classification task was solved, and its performance was evaluated using four key metrics: Accuracy (Eq. (3)), Precision (Eq. (4)), Recall (Eq. (5)), and F1-score (Eq.(6)). The true positive (TP) represents instances where y_i was accurately predicted as y_i; true negative (TN) corresponds to cases where y_j was correctly identified as y_j; false positive (FP) refers to instances where y_j was erroneously predicted as y_i; and false negative (FN) denotes cases where y_i was inaccurately predicted as y_j.

(3) $$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(4) $$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(5) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(6) $$F1-score={\displaystyle \frac{2\times precision\times recall}{precision+recall}}$$

For our evaluation, we have used precision, recall, and f1-score values. The results are averaged from three runs and presented in Table 6.

Results of Task 2

As elaborated in the Methods & Materials section, we assessed the performance of our text generation task using two key metrics: BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2001) and ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence) (Lin, 2004). These metrics are commonly used to evaluate the quality of generated text compared to reference texts.

The BLEU score calculates the geometric mean of modified n-gram (N = 4) precision (P_n) with a brevity penalty (BP) to account for differences in length between the generated and reference texts:

(7) $$BLEU=BP\times exp\left(\sum \left({\displaystyle \frac{\log \left(Pn\right)}{N}}\right)\right)$$where N is the maximum order of n-grams considered, the brevity penalty (BP) is defined as:

(8) $$BP=\{\begin{array}{cc}1& ifc>r\\ e^{\left(1-r/c\right)}& ifc\le r\end{array}$$where r is the total length of the reference texts, and c is the total length of the generated texts.

The ROUGE-L score measures the longest common subsequence (LCS) between the generated text and the reference text, taking into account recall and precision. The ROUGE-L score is calculated as follows:

(9) $$ROUGE-L={\displaystyle \frac{\left(1+{\beta }^2\right)\times LCSPrecision\times LCSRecall}{\left({\beta }^2\times LCSPrecision+LCSRecall\right)}}$$

where β = 1 is a weighting factor that adjusts the importance of precision relative to recall. LCS Precision is the ratio of the length of the LCS to the length of the generated text, while LCS Recall is the ratio of the length of the LCS to the length of the reference text.

The results are presented in Table 7.

Data processing methods for the EPO/USPTO dataset

This section delineates the methodology and steps employed in constructing a dataset comprising synthesis procedures, their structured counterparts, reactants, and products represented in IUPAC and SMILES notations, similar to the D. M. Lowe’s dataset. Given the dataset’s potential value for chemists and AI researchers, describing the process in detail is crucial. The procedure extends beyond conventional text preprocessing techniques and requires more steps and data preparation methods. The two main tasks analyzed in the article are included in the description, emphasizing that the outlined steps serve as one possible approach, which may be adapted or enhanced to align with the specific objectives of researchers. Each step is also finished in such a manner that one would be able to continue without starting at the first one to facilitate the research process. Throughout our work, we utilize open-source data and code and provide access to our resources.

From raw patent data to organic chemistry patents

The data utilized in this study originate from patent records because of their availability and structured format. Among the various patent databases available, the United States Patent and Trademark Office (USPTO) and the European Patent Office (EPO) databases were chosen for this research. The database selection criteria included: the database size, accessibility in bulk and freedom of usage, and the absence of any costs associated with accessing, using the data or license issues. This ensures that other researchers can replicate our study using the same data without incurring additional costs. Other notable patent databases include the World Intellectual Property Organization (WIPO) Patent Collection, The Japan Patent Office (JPO) Database, The China National Intellectual Property Administration (CNIPA) Database, and The Korean Intellectual Property Office (KIPO) Database. However, some may have specific limitations or costs associated with accessing bulk data.

The US Patent and Trademark Office (USPTO) makes a database of patent applications and grants available via its bulk data storage system (https://bulkdata.uspto.gov, accessed on 8 of March 2022), which can be accessed free of charge. This database covers patents dating from 1971 and includes approximately 8.2 million patents with a total size of approximately 700 GB. Utilizing scripts or software to aid in the downloading process is recommended. The data is available for use, reuse, and distribution under the terms outlined in the USPTO’s Open Data statement (https://developer.uspto.gov/about-open-data, accessed on 8 of March 2022).

The European Union (EU) Patent Database is hosted on the Google Cloud Platform (GCP). Information on downloading the database and a user manual can be found on the European Patent Office’s website (https://www.epo.org/searching-for-patents/data/bulk-data-sets/text-analytics.html, accessed on 8 of March 2022). While the data is available at no cost, there are charges associated with transferring it from GCP to local storage. The database includes a total of 3.7 million patents in English, with a total size of approximately 255 GB. Earliest patents in the database date back to 1978. The data is available under the Creative Commons Attribution 4.0 International Public License.

In order to utilize the patent data in a programming environment, it was first necessary to convert the data into a format that could be processed. While the European Patent Office (EPO) data set is in a uniform format and structure, the United States Patent and Trademark Office (USPTO) data set employs a variety of formats. Therefore, specialized scripts were written for each format to facilitate subsequent processing steps. The second major step involved extracting patents that pertained to a specific classification tag of organic chemistry. The USPTO data set includes classification identifiers, while the EPO data set does not. To overcome this, the EPO’s freely available Application Programming Interface (API) was utilized to collect the classification data separately, resulting in a list of patent IDs with the organic chemistry classification tag. After this step, 230,594 EPO patents and 511,959 USPTO patents remained for further analysis. It is recommended for researchers to start with the dataset created after this step because it is significantly smaller, and the progress up to this step has been just the collection of data by straightforward programming algorithms.

From organic chemistry patents to procedure paragraphs

We developed a methodology for classifying individual paragraphs within patterns to separate chemical synthesis procedures from other texts. This approach leverages deep neural networks and has been specifically designed by our research team for this purpose (referred to as Task 1). We successfully classified 182,315,235 paragraphs through this process, identifying 3,564,546 as relevant synthesis procedures after removing duplicate procedures. The paragraphs were additionally processed, removing unsuitable paragraphs that reference other procedures or do not contain full names of reactants or products.

Subsequently, we converted each paragraph into a more accessible format using a model developed during our research (Task 2). This new format facilitated more efficient processing, allowing easy access to distinct actions, action parameters, and chemical name extraction. Each paragraph was split into sentences using the spaCy library (Honnibal & Montani, 2017) and transformed the text into a structured format, which could then be reconstructed into a complete paragraph. As a result, our dataset comprised 3,058,295 unique synthesis procedures.

From procedure paragraphs to the EPO/USPTO dataset

Structured paragraphs were used to extract the names of reactants and products in chemical synthesis processes. Reactants were identified through the ADD and MAKESOLUTION actions, while the YIELD action denoted the product. The extracted compound names were either in the IUPAC nomenclature or had trivial naming. These actions were chosen because they represented the addition of reactants, while the action YIELD provided access to the products.

A multi-step process was developed to convert all compound names SMILES notation, which enables compound and substructure searching and is compatible with popular chemical programming libraries. Three translation methods were identified as helpful for this task based on their accuracy and coverage: (1) the PubChem application programming interface (API), which provides a robust platform for translating compound names in both trivial and IUPAC forms (Kim et al., 2022); (2) Open Parser for Systematic IUPAC Nomenclature (OPSIN), a versatile open-source name-to-structure conversion tool that excels in handling complex IUPAC names (Lowe et al., 2011); and (3) the ChemAxon application Instant JChem, a robust structure database management tool that supports various naming conventions (http://www.chemaxon.com, accessed on 5 December 2022).

The chemical entity detector ChemDataExtractor 2.0 facilitated the process due to its compatibility with Python, extensive documentation, and GPU-accelerated usage (Mavračić et al., 2021). This tool scans text and returns chemical entities, specifically compound names. These features were crucial in efficiently processing large volumes of text while minimizing errors. Using ChemDataExtractor 2.0, a comprehensive list of all compounds was generated from procedure paragraphs in the dataset. Subsequently, a dictionary mapping IUPAC or trivial names to SMILES was created using the three translation methods. This approach increased the total number of translated names, as each method individually translated only about 60 to 70% of the input names. No known solutions exist for IUPAC to SMILES translation other than manual methods, which are infeasible for a large vocabulary of 3,709,454 items. The generated vocabulary was employed to convert the names of reactants and products to SMILES notation and are put into separate columns in the final dataset. Lastly, common solvents (e.g., water or ethyl acetate) were removed from the reactant column, as they occasionally appeared in the ADD or MakeSolution steps and served only as the synthesis medium. This step ensured that only the reactants and products remained, rather than various solvents facilitating the synthesis and the work-up process. To enable further research and investigation, we've made our final dataset and the source code for our extraction and transformation scripts available to the public. The dataset, in a machine-readable format, and the Python scripts can be accessed from our project repository on GitHub (https://github.com/Mantas-it/ActionExtraction). Description and examples are provided within the repository to facilitate the understanding and utilization of both the dataset and the models.

The dataset encompasses a comprehensive range of organic compounds with varied structural characteristics. These include small organic molecules, organometallics, and biomolecules, each with distinct functional groups, including but not limited to, carbonyls, amines, alkenes, alkynes, and aromatic systems. Each of these functional groups can partake in various types of organic reactions such as addition, substitution, elimination, and rearrangement reactions, providing a broad coverage of synthetic pathways. These compounds participate in an assortment of organic reactions, reflecting a comprehensive representation of the reaction landscape in organic synthesis. From foundational reaction types such as substitution, addition (electrophilic, nucleophilic), and elimination, to more advanced and specialized reactions such as pericyclic reactions, cross-coupling reactions and photochemical transformations, our dataset captures a spectrum of mechanistic possibilities.

Task 1: overall results

Table 6 presents the results for Task 1, where a BiLSTM classifier with a vocabulary size of 16,000 words achieved the highest accuracy of 0.939. This accuracy significantly surpasses the random (accuracy-0.553) and majority (accuracy-0.664) baselines, demonstrating the method’s suitability for the solving task. Classifiers with a vocabulary size of 10,000 underperformed, likely due to a lack of expressiveness or the presence of out-of-vocabulary words. On the other hand, classifiers with vocabulary sizes larger than 16,000 did not yield better results due to data sparsity, as the probability of observing specific words decreases when the vocabulary size increases. It is worth noting that the dataset utilized for this task is considerably smaller than those commonly used in NLP tasks, which typically consist of millions of examples. The superiority of the BiLSTM model can be credited to its distinctive architecture that effectively utilizes the input in two directions, specifically from-past-to-future (forward) and from-future-to-past (backward) information. This bidirectional context assimilation is critical in making sense of complex chemical procedures where dependencies can exist in either direction. The application of LSTM within this architecture contributes significantly to the model's effectiveness. Firstly, LSTMs ensure more stable learning by controlling the flow of gradients throughout time, which alleviates the prevalent vanishing/exploding gradient problem. This feature is particularly important in capturing long-term dependencies across words and sentences, a common characteristic in scientific text. Secondly, LSTMs are designed to be more robust to noise or incomplete data. Their memory cells selectively retain, or discard information based on the determined importance of the input. This capability allows them to filter out noise and accurately capture the information conveyed in the data. Lastly, LSTMs’ inherent capacity to retain or forget information as required, serves as a critical function in discerning the intricate details embedded in chemical procedures. The architectures of models can be found in the project’s repository (https://github.com/Mantas-it/ActionExtraction).

A direct comparison of our results with previously reported findings is not feasible, as we employed a custom training dataset for this task. However, we also compared our approach with traditional ML methods such as naïve Bayes (accuracy-0.810, precision-0.753, recall-0.799, F1-score-0.768) and support vector machines (SVM) (accuracy-0.885, precision-0.837, recall-0.882, F1-score-0.855), both of which yielded considerably high results (especially SVM) but significantly worse compared to our best BiLSTM model. Although the optimal BiLSTM classifier’s accuracy is relatively close to traditional ML methods, improving the scores further remains challenging due to the difficulty in distinguishing similar paragraphs. An essential metric to consider is recalled, as high scores in other metrics coupled with a low recall could indicate a substantial proportion of paragraphs being falsely identified as procedures. In this context, it is preferable to miss actual procedures rather than have various paragraphs mixed in, as this would complicate downstream processing when preparing the dataset.

With their attention layers, transformer neural networks are capable of handling long-term dependencies, making them seemingly ideal for tasks involving paragraph inputs. However, as observed in Table 6, the performance of transformer networks ranks second best. Notably, during the training process, these networks exhibited rapid overfitting. One potential explanation for this phenomenon is that the training dataset might be too small for transformer neural networks to train effectively, given that they possess a substantially larger number of weights than the other classifiers tested in our research. Considering the superior performance of transformer networks in numerous other NLP tasks, it can be hypothesized that a larger dataset would enable transformers to outperform other classifiers. Besides, instead of training our transformer models, existing pre-trained ones (e.g., BERT or sentence transformers) can be incorporated and later fine-tuned for our classification problems. This research direction is in our immediate plans. Despite it, there is always a risk that the available models may have too little chemistry knowledge.

Task 1: error analysis and important observations

When analyzing the mislabeled instances, we found that the top-performing BiLSTM classifier can effectively differentiate between synthesis procedures and other paragraphs with high confidence. However, most errors occur when synthesis paragraphs deviate from the expected format. As outlined in section describing the data, paragraphs should contain full names of products and reactants without references to other procedures or compounds. The task’s complexity increases when phrases such as title compound or the desired product are used with the product name. While the classifier can handle such cases to some extent, it is not consistently successful. Rare instances where title compound is followed by the full compound name—provided the remainder of the paragraph is reasonable—require nuanced classification. Increasing the dataset size with more instances could potentially enhance accuracy scores. In practical terms, the existing errors are somewhat mitigated during further processing of identified procedures when synthesis paragraphs containing specific key phrases are excluded from the resulting database.

The top BiLSTM classifier is approximately 3–5 times faster when used in an interface mode than the top transformer classifier, allowing it to classify the current dataset of 170 million paragraphs in about 12 h using a single GPU. Processing speed is crucial, as the next potential iteration of this task involves developing a similar methodology for open-source chemistry research articles to source more synthesis procedures, likely necessitating the processing of a far greater number of paragraphs than those available from patents.

Task 2: overall results

Table 7 describes the results for Task 2, in which a transformer with a vocabulary size of 16,000 words attains the highest BLEU score of 0.951. As observed in the first task, a vocabulary of 16,000 words allows models to achieve the best performance. LSTM and BiLSTM models are less suitable for this task, as the optimal BiLSTM model only achieves a BLEU score of 0.671, significantly lower than the transformer model’s score. Table 7 does not include T5 small and base test scores with any other vocabulary sizes other than the 32,000 vocabulary size column because a pre-trained T5 model and its tokenizer, which has a vocabulary of 32,128, were utilized. The fine-tuned T5 models produced BLEU scores comparable to the best model (T5 small–0.906, T5 base–0.902). Although the T5 model is considered more advanced for NLP tasks, its pre-trained status on a large corpus with limited relevance to the task at hand may not provide significant advantages. In contrast, transformer networks trained from scratch can efficiently learn to translate unstructured synthesis procedure text into a structured format. Transformers outperform other models mainly due to their unique architecture that enables the capture of dependencies regardless of their distance in the text, courtesy of the self-attention mechanism. This feature is particularly crucial in deciphering scientific language, where the context is often dependent on non-local interactions within the text. Moreover, the transformer model does not necessitate the sequential processing of data, which allows for better utilization of parallel computing resources, consequently enabling efficient model training. Also, the multi-head attention within transformers is able to focus on different parts of the input sequence while translating, allowing the model to handle varying contexts effectively, thus enhancing the quality of translation. These results are comparable to scores achieved by a transformer model (BLEU–0.850) (Vaucher et al., 2020) and a fine-tuned T5 small model trained for multiple chemistry-related tasks (BLEU–0.953) (Liu et al., 2019). It is important to note that our study employed a distinct dataset specifically created for this task, which includes additional parameters for one step, as described in the data section. The achieved high BLUE value encourages us to experiment with this problem even more. Input texts can be modified (paraphrasing, summarizing, back-translating, or by hand) by making them even less formalized and, therefore, better comprehensible for non-chemistry specialists. On the other hand, the output can become even more formalized and understandable to robots. However, the existence of such a method could bring chemistry to a broader circle of people (especially in the education sector). Thus, the gap between the level of formalization between the input and the output would become much larger than it is right now, and therefore the solving problem would become even more challenging. Nevertheless, it is an exciting future research direction.

Task 2: error analysis and important observations

The optimal model for this task attains a BLEU score of 0.953 and generally makes very few errors. Accuracy has been evaluated by measuring how many sentences are predicted identically, as they are the testing dataset, and it achieved a score of 0.823. The most common mistakes involve minor details, such as the order of additional actions when multiple components are added successively. In some cases, the model fails to detect the amount of the added compound and does not include it in the structured sentence format. Errors mainly occur with the actions InvalidAction and FollowOtherProcedure, which are highly specific and infrequent. It was possibly not provided with enough examples for the model to learn each case effectively. InvalidActions refers to unfeasible actions, such as when a sentence describes a reaction mechanism, while FollowOtherProcedure denotes a step where another procedure or method is referenced in a patent. Moreover, certain sentences only partially describe a step and then reference a similar step in other procedures, making translation challenging. The distinction between these two actions can be subtle and difficult for an expert to discern during manual dataset labeling. In large-scale data processing, it is advisable to exclude paragraphs with InvalidAction and FollowOtherProcedure actions, as these include either incorrect operations or content that surpasses the model's comprehension, thereby undermining the ability to generate sensible translations. In the range of available procedure sources, these specific actions are encountered in approximately 3 to 7% of all instances. Conversely, in an experimental laboratory context where procedure processing is conducted on a smaller scale and manual verification of translations is feasible, it becomes beneficial to conduct thorough examinations of these instances. This is because not all occurrences of InvalidAction and FollowOtherProcedure are necessarily detrimental to the structure of the converted procedure. Most crucially, no instances of incorrectly converted compound titles have been observed.

The fine-tuned T5 small model attains a lower BLEU score of 0.906. The tokenizer used with the pre-trained T5 model can be fine-tuned by training it on a task-specific corpus and adjusting the vocabulary to include unique characters and words that might appear. Although such tests were conducted, the model achieved even lower scores (BLEU-0.780). This outcome may be attributed to the tokenizer significantly altering the vocabulary when trained on synthesis procedures, with only around 2,000 tokens out of 30,128 remaining the same. This drastic change likely necessitates training the original embedding weights alongside the rest of the model. Consequently, the model does not benefit from pre-training and would need to be trained from scratch on a large chemistry text corpus, which is beyond the scope of this article.

The fine-tuned T5 model can accurately translate unstructured procedure sentences with minimal errors. Most of the mistakes resemble those previously described, occasionally including incorrect quantities, such as copying the original expression from the sentence as (about 98 mg) instead of (98 mg). However, it is worth noting that the model can still be employed for applications in contexts where procedures are relatively simple and devoid of complex or misleading information or expressions. Furthermore, the T5 model is used with the Python library HuggingFace, which offers convenient tools for setup, interface, and integration compared to transformer models trained with OpenNMT, which require several steps for setup. However, the T5 model is significantly larger, and the translation process is slower than the transformer models.