Learning-based short text compression using BERT models

Learning based compression

In recent years, many learning-based data compression methods have been developed that work much slower than traditional compression methods but can offer higher compression ratios.

Cmix, one of the learning-based data compression methods developed by Byron Knoll, has three different stages. The preprocessing stage transforms the input data into a more compressible form. The model prediction stage uses more than 2,000 independent models specialized for certain data types such as text, executable files or images. The context mixing stage includes the byte level LSTM mixer and the bit level context mixer similar to the PAQ8. The output of the context mixer is refined using SSE (secondary symbol estimation) algorithm. Knoll created two other compressors using only LSTM: lstm-compress and tensorflow-compress.

NNCP, developed by Fabrice Bellard, is a lossless data compression method that uses neural network models based on LSTM and Transformer architectures (Bellard, 2019). In the tests of the first version, LSTM gave better results than Transformer. In the second version developed two years later, Bellard used only the Transformer architecture and significantly improved its performance (Bellard, 2021). Bellard also developed a program called gpt2tc, which compresses English texts or performs auto-complete on them using the GPT-2 model.

In recent years, there have been significant advancements in utilizing transformer-based models and large language models (LLMs) for various applications. For instance, a transformer-based method, GPTZip, performs compression using the GPT-2 model (Nesse & Li, 2023). In this study, to assess its effectiveness on shorter or multilingual texts, the GPTZip is integrated into experimental evaluations.

Sentence compression is the process of producing a shorter sentence by removing some unnecessary words while preserving the grammar and important content of the original sentence. Sentence compression is actually a type of lossy text compression. Although there are some sentence compression studies using BERT (Niu, Xiong & Socher, 2019; Nguyen et al., 2020; Park et al., 2021; Jun, 2020), no study has been found in the literature in the field of lossless text compression.

Short text compression

There are simple and fast methods developed to compress short texts, most of which just use a static dictionary. Smaz is a simple method developed by Salvatore Sanfilippo that can compress short texts quickly and at good ratios. It compresses English text better than other languages, as it contains a static dictionary specific to English text written in lowercase letters. It can also compress HTML and URLs, as the dictionary also includes entries such as “http://” and “ </”.

Shoco is a method written in C language by Christian Schramm to compress short texts. The default compression model is optimized for English words, but another compression model can be created with different input data. Shoco is an entropy encoder and generally gives worse compression ratios than dictionary-based Smaz when compressing English texts. However, if there are many words in the text that are not included in the dictionary, such as numbers, Smaz can expand the text. Shoco, on the other hand, never expands the size of ASCII type texts (it can increase non-ASCII characters by two times).

The Short Message Arithmetic Compressor (SMAC) uses a word-based static dictionary, 3rd order letter statistics and arithmetic coding to compress short text messages such as SMS and Twitter (Gardner-Stephen et al., 2013). It is slower than Shoco and Smaz; however, it provides higher compression ratio. It offers full Unicode support and accepts UTF-8 strings. SMAC never increases the length of a string during compression, and uncompressed strings can be passed to the decompressor.

b64pack is another algorithm developed for the compression of short text messages (Kalajdzic, Ali & Patel, 2015). In the first stage of the algorithm, the input text is converted into a format that can be well compressed in the second stage. The second stage consists of a transformation that reduces the size of the message by a fixed fraction of its original size. b64pack is a simple algorithm that can only be used for English texts. It is fast but offers a moderate compression ratio.

In Aslanyürek & Mesut (2023), static dictionaries are created with machine learning for different languages and different topics. The compression method determines which static dictionary is more compatible with the text to be compressed and compresses it with that dictionary (Aslanyürek & Mesut, 2023). Although this method is slower than the others, better compression ratios can be obtained due to the use of different static dictionaries.

Word based compression

Word-based text compression methods are techniques where symbols are selected as words. By substituting frequently occurring words with shorter symbols or codes, the overall size of the text can be reduced. This approach is particularly useful in applications where efficient search and retrieval of information are crucial, such as in databases and information retrieval systems. While these methods may not achieve the highest possible compression ratios, they offer a balanced trade-off between compression efficiency and the ease of performing operations like searching and indexing on the compressed text. In these methods, the symbol substitution process is applied to words, and sometimes the primary goal is not maximum compression but rather indexing and searching within the compressed text.

End-Tagged Dense Code (ETDC) and (s,c)-Dense Code (SCDC) are word-based semi-static compression methods that are effective when searching for a single word in a compressed text (Brisaboa et al., 2007). Dynamic ETDC (DETDC) and Dynamic SCDC (DSCDC) are better than semi-static ones in terms of compression speed, but they are worse in terms of decompression speed (Brisaboa et al., 2008). Dynamic Lightweight ETDC (DLETDC) and Dynamic Lightweight SCDC (DLSCDC) are able to perform decompression as fast as semi-static dense codes (Brisaboa et al., 2010). Although the compression speeds of dynamic lightweight codes are worse than dynamic dense codes, they are better than semi-static dense codes. On the other hand, dynamic lightweight dense codes are not better than dynamic dense codes, and dynamic dense codes are not better than semi-static dense codes in terms of compression ratio.

The Tagged Word-Based Compression Algorithm (TWBCA) is another word-based semi-static compression method that allows searching for a word or a phrase in compressed text (Buluş, Carus & Mesut, 2017). Although TWBCA is slow at decompression, it is better than ETDC in compression speed.

Multi-stream word-based compression algorithm (MWCA) is a method similar to ETDC, but differs from ETDC in that it produces multi-stream output (Öztürk, Mesut & Diri, 2018). Exact word matching can also be done on compressed texts with MWCA. Although the compression ratio and decompression speed of MWCA are worse than ETDC, the compression speed is similar to ETDC, and the multi-stream output structure of MWCA allows much faster compressed search than ETDC.

Dataset used for compression

For fine-tuning the small models, a 20 MB of text is extracted from the English 50 MB file which is in the Pizza&Chili corpus, and sentences are obtained from this subset. Subsequently, datasets are obtained from Huggingface for six different languages, 20 MB of text is extracted from each of them to create multilingual training data. Datasets taken from Huggingface are named as “germanquad” (Möller, Risch & Pietsch, 2021), “Spanish speech text”, “xnli2.0 train French”, “Italian tweets 500k”, “Dutch social” and “Turkish instructions” respectively. Emojis, punctuations and links are removed from texts and obtained data is used for training.

The datasets consist of raw text rather than structured content. To enhance the prediction accuracy of the models in the desired language, whether in English or multilingual, raw texts in natural language are collected and used for fine-tuning. At this stage, apart from the tokenization performed by the BERT models, no additional feature selection or extraction is applied.

The obtained sentences are used for training the selected TinyBERT models. Before training, the dataset for multilingual and English data is combined and shuffled. During the training process, to prevent possible overfitting, the dataset is divided into training and validation sets as 90%–10%, and the loss values for both cases are obtained. The datasets used, along with their sizes and training-validation ratios, are provided in Table 1.

Models used for compression

MLMCompress allows the use of any masked language model for compression. Compression ratio of MLMCompress is highly correlated with prediction accuracy of masked language model. There are several options available for improving prediction accuracy. One of them is to use a larger model, but this could result in slower compression due to slower word prediction phase. Another option is to enhance prediction performance by fine-tuning a smaller model, thus achieving higher compression and decompression speeds.

There are fine-tuned versions of BERT models for different languages in Huggingface. These models are available as cased and uncased. In this study, the following four different cased models, two of which are large and the other two are small, are used:

• Model A: bert-base-cased / Large / English

• Model B: bert-base-multilingual-cased / Large / Multilingual

• Model C: tzyLee/quant-tinybert / Small / English

• Model D: dbmdz/bert-tiny-historic-multilingual-cased / Small / Multilingual

While large-sized models are used with MLMCompress without any preprocessing, the small-sized models are fine-tuned and then used in tests. The training parameters of the models are provided in Table 2.

The models are trained on the dataset for 100 epochs. Since the models are designed for short text compression, the maximum token length is set to 50. Given that compression will be performed in different languages, and especially in agglutinative languages where different suffixes necessitate the suggestion of different words, the whole word mask parameter is set to true. The mask probability parameter, which indicates the likelihood of a word being masked in a given sentence during training, is left at its default value. The training and validation loss values obtained during the training process are presented in Fig. 2.

As shown in Fig. 2, the variation in the loss values of the models is very small between 80-100 epochs. Therefore, the training is concluded in 100 epochs. In the final state, the loss value for tzyLee/quant-tinybert is obtained as 0.081, and the validation loss value is 0.06. For the dbmdz/bert-tiny-historic-multilingual-cased model, the loss and validation loss values are obtained as 0.7638 and 0.7620, respectively.

The compression and decompression methods using this model are described in the following subsections, respectively.

Compression method

MLMCompress starts the compression process using the maximum window size and prediction count parameters. Maximum Window Size refers to an array of words of a size that will be given to BERT for making predictions. Since the compressed word is encoded as one byte and two escape characters are required for uncompressed words and empty strings obtained at input splitting process, a maximum of 254 is used as the prediction count parameter.

In the first step of compression, the input is split into words W[w₀, w₁, …, w_m]. At first, w₀ is added to the R array for proper decompression. After this step, starting from w₀ each window for prediction is prepared with a [MASK] token in the end and added to an array named I. Respective words will be added to the window until the window size reaches maximum window size (n). When the window size reaches n, the window update process is performed by removing the first word from the window and adding the next word to the window at each stage. The reason for using an array like I instead of making predictions in the first place is to reduce the time cost of the prediction phase of BERT. BERT can make predictions for batch inputs as a whole and efficiency is gained by pre-creating all windows to be predicted and providing them to BERT in a batched manner.

After obtaining Predictions from BERT, r predictions are requested for Window[PC] as Predictions[PC] {p₁, p₂, …, p_r}. If the masked word exists in the prediction list, its index value is written into the C array. If the word is not in the Predictions[PC], the escape character ‘0’ is written into the C array and word is written into the R array.

After the creation of the arrays is completed, the R array containing words is converted into a string by combining space characters between the words and is encoded with Static Huffman Code Tables. If a small text is tried to be compressed with the Dynamic Huffman method, expansion will occur rather than compression, since the coding table obtained from the Huffman tree must also be sent to the decoder. For this reason, Static Huffman Code Tables are created for seven different languages used in the tests and these tables are used in both encoding and decoding. Finally, the C and Huffman encoded R arrays are concatenated and written to the output (compressed) stream. The steps of the compression process are given in Algorithm 1 .

 
_________________________________________________________________________ 
 
Algorithm 1 Compression algorithm of MLMCompress 
  1:  W = split input text by spaces and punctuations 
  2:  Window = [W[0]] 
  3:  R = [W[0]], C = [], I = [] 
  4:  for i=1 to Length(W) do 
 5:     if W[i] is not empty then 
 6:         Add a list [Window, [MASK]] to I 
 7:         Add new word to Window 
 8:         if Length(Window) = n {Max window size} then 
 9:            Remove Window[0] 
10:         end if 
11:     end if 
12:  end for 
13:  r as prediction size 
14:  Predictions = BERTPredsForAllWindows(I,r) 
15:  PC = 0 as prediction counter 
16:  for i=1 to Length(W) do 
17:     if W[i] is empty then 
18:         Append 1 byte to C 
19:     else 
20:         if W[i] is in Predictions[PC] then 
21:            Ind = index of W[i] in Predictions 
22:            Append Ind+2 to C 
23:         else 
24:            Append 0 to C 
25:            Append W[i] to R 
26:         end if 
27:         PC = PC + 1 
28:     end if 
29:  end for 
30:  return  Concat(C, Huffman(R)) 
_______________________________________________________    

An example for compression of the sentence “He feels better since it is a sunny day” is given in Table 3. It is assumed that BERT makes 3 predictions (r = 3) and the maximum window size is 2 (n = 2). In the first step, the word “He” is given to BERT and predictions are taken from it. Since BERT predicts the word “feels” as the 3rd word, the value ‘3’ is coded into the C array and the window is updated by adding the word “feels”. In this step, the first word “He” is also written into R array for decompression. In the next step, the window “He feels” is given to BERT and the predictions are received. Since BERT cannot guess the word “good”, a ‘0’ byte is added to the C array and the word “better” is written to the R array. Since the window size is 2, the word “He” is removed from the window and the word “better” is added to the window after this stage. Since the word “since” is predicted as the 1st word by BERT, the index 1 is written to the C array. The process is applied to all remaining words using these three rules. After the R array has gone through the Huffman coding process, the C and R strings are combined and written to the output stream.

Decompression method

In the first step of decompression, the C and Huffman encoded R arrays are separated and the unencoded R array is obtained by Huffman decoding. After this step, the first word is taken from the R array and the index values are read from the C array. At each stage, if the next index value is not ‘0’, prediction is performed by giving the window and r to BERT. After the predicted words are taken from BERT, the word from Predictions array at the index value which is read from the C array is added to the output stream. If the index value is ‘0’, the next word from the R array is added to the output stream. After this step, if the window size is less than n, this word is also added to the W. If the window size is equal to n, the first word in the window is removed and the last word is added to the end. When the process is complete, the output (decompressed) stream is obtained. The creation processes of the output stream are given in the Algorithm 2 .

 
__________________________________________________________________________ 
 
Algorithm 2 Decompression algorithm of MLMCompress 
  1:  C, HuffR = Split(Compressed) 
  2:  R = HuffmanDecode(HuffR) 
  3:  D = R[0] + ” ” 
  4:  W = [R[0]] 
  5:  RCounter = 1 
  6:  CCounter = 0 
  7:  while CCounter < Len(I) do 
 8:     if C[CCounter] = 0 then 
 9:         Append R[RCounter] + ” ” to D 
10:         Refresh Window with adding R[RCounter] 
11:         RCounter = RCounter + 1 
12:     else if C[CCounter] = 1 then 
13:         Append a space to D 
14:     else 
15:         Predictions = BERTPredsForWindow(W) 
16:         Word = Predictions[C[CCounter]-2] 
17:         Append Word + ” ” to D 
18:         Refresh Window with adding Word 
19:     end if 
20:     CCounter = CCounter + 1 
21:  end while 
22:  return  D 
_____________________________________________________________________________________    

An example for the decompression method is given in Table 4. To facilitate easy tracking of the next element, the processed elements are removed from the table rows. In the algorithm’s operation, these values are read by incrementing the CCounter value over the list. After the C and R arrays are obtained, the first word in the R array “He” is written into the window and output stream and predictions are taken from BERT. Since the value ‘3’ is read from the C array, the 3rd prediction is taken from the prediction list and written to the output stream. After the window is updated to “He feels”, CCounter will be 1 and the next value ‘0’ is read from the C array. Since the value ‘0’ means that BERT could not predict the word, the next word from R is written to the output stream without using prediction procedure. Since the window size is 2 for this example, the word “He” is removed from the window and the word “better” is added to the window. In the next step, since the last value in C is “1”, the 1st BERT prediction made for the “feels better” window, the word “since”, is written to the output stream. The process is applied to the end of C and R arrays and the decompressed stream is obtained.

Compression ratio results

MLMCompress is tested with different window sizes (n = 5, 10, 15) and different prediction sizes (r = 64, 128, 254). The compression ratio results of the tests performed with four different models on English and Multilingual short texts are given in Table 5.

As seen in Table 5, most of the times both for English and Multilingual texts, combination of w = 15 and p = 254 gives the best compression results for all different length ranges and models. Therefore, for comparison with other compression algorithms, w = 15 and p = 254 results are used for all models.

The compression ratio results of the English short texts for different methods are given in Table 6. As can be seen in the table, MLMCompress Model A achieved the best compression ratios across all size ranges. NNCP, which is another learning-based method in our test and compresses the 1GB ‘enwik9’ text data at the highest rate today (based on Large Text Compression Benchmark results in November 2023), expands short texts instead of compressing them. Since NNCP default cannot compress most texts larger than 500 bytes and NNCP lstm cannot compress most texts larger than 600 bytes, results regarding these sizes are not given.

According to the test results, the method with the best compression ratio among the NNCP versions is lstm-fast, and the method with the worst compression ratio is enwik8. While Gzip cannot compress texts smaller than 100 bytes, the compression ratio increases significantly as the text size increases, similar to NNCP. Gzip gave similar results with the short text compression methods Shoco and Smaz in the size range of 300-399 bytes, and started to give better compression rates than them for texts larger than 400 bytes.

The compression ratio results of the tests performed on multilingual short texts are given in Table 7. Similar to the English results, the best compression ratios in every size range for multilingual texts are obtained with MLMCompress that uses a large model (Model B), and the worst compression ratios are obtained with NNCP enwik8. As seen in Tables 6 and 7, MLMCompress achieves worse compression ratio results with tiny models (Model C and Model D). This is because larger models have a higher prediction accuracy, which allows for an increase in the number of compressed words. GPTZip achieves up to 20% better compression than MLMCompress in the best scenarios, this advantage diminishes with multilingual files.

Gzip and all versions of NNCP gave similar results in both tables regardless of the language of the text. However, Smaz and Shoco methods compressed multilingual texts with a much worse compression ratio than English texts due to their static dictionary structures specific to English. While the compression ratio increases significantly as the size of the texts increases in NNCP and Gzip methods, the effect of text size on the compression ratio is much less in short text compression methods.

In MLMCompress, Model A’s compression ratio for English short texts is higher than Model B’s compression ratio for multilingual short texts. However, the opposite is the case for Model C and Model D. Although the compression ratio improves as text size increases in MLMCompress, the difference is not as much as in NNCP and Gzip.

Compression and decompression speed results

Learning-based compression methods perform compression and decompression much slower than traditional methods. For this reason, the MLMCompress method is compared in this section only with NNCP. Compression and decompression speeds in nine different size ranges are given in Table 8 for English texts and in Table 9 for multilingual texts. In these tables, as in the previous section, w = 15 and p = 254 results are used for all MLMCompress models. This configuration, which offers the highest compression ratio, is actually the slowest in terms of compression and decompression speeds.

While NNCP’s compression and decompression speeds increase as text size increases, in MLMCompress text size has little effect on compression and decompression speeds. As seen in Tables 8 and 9, NNCP’s compression and decompression speeds are very close to each other in all versions.

In MLMCompress, compression is faster when large models are used and decompression is faster when small models are used. The reason why decompression is faster in smaller models is that less decompression is done with BERT since fewer words can be compressed. MLMCompress Model A is 4–6 times slower in compression and 25–30 times slower in decompression than MLMCompress Model C. Similarly, Model B is 7–10 times slower than Model D in compression and 19-23 times slower in decompression.

The unexpected result about NNCP is that the lstm-fast version performs compression and decompression operations about two times slower than lstm. Since the compression ratio of lstm-fast is also better than lstm, probably the lstm and lstm-fast parameters may have been assigned inversely in the Python library we use. The fastest NNCP version, default, is approximately 16% faster in compression and approximately 25% faster in decompression than lstm. NNCP enwik8, which gives the worst results in compression ratio, is also very slow compared to other methods in compression and decompression.

As noted above, although GPTZip can produce better results than MLMCompress, these differences become significantly more pronounced when considering the speed difference. In this context, MLMCompress demonstrates a balanced performance in terms of both compression ratio and speed. This balance can be attributed to the size of the model and the number of parameters used.

To ensure the accuracy of the results, sentences of varying lengths are extracted from the LAReQA dataset (Roy et al., 2020) using a step size of 100, similar to the previous dataset. The compression ratio, compression speed, and decompression speed results for these sentences are calculated. The results are presented in Table 10.

Upon examining the data presented in Table 10, it is evident that GPTZip can produce better compression results than MLMCompress. However, when the speeds at which these results are achieved are considered, the difference becomes strikingly significant. For instance, while producing faster results, GPTZip compresses files 6.3% more effectively than MLMCompress in its best case, yet remains 10 times slower in terms of speed.