Automatic detection of semantic primitives using optimization based on genetic algorithm

Introduction

There are plenty of dictionaries that are available online oriented at human readers. Moreover, publicly available dictionaries are especially useful in various computational linguistics (CL) problems (for example, word definition generation, word sense disambiguation, etc.). In many of these tasks, the meaning of words is a key feature. Furthermore, dictionaries provide useful information about relations between words and how they can be expressed one through another.

On the other hand, traditional dictionaries have a problem of cycles. The problem is described as follows. Each word in the vocabulary should be defined. Furthermore, each word in the definition should be defined and so on. It means that the process is either endless or, at some point, the cycle is created. For example, here is what the cycle can look like: the word “pact” can be defined via the word “agreement”, the word “agreement” can be defined via the word “treaty”, and the word “treaty” can be defined via the word “pact”. In dictionaries, these cycles are usually long (they include many words), which is a good thing for human readers, because eventually, they encounter the word that is familiar to them.

This brings us to the linguistic concept of semantic primitives (SP, sometimes also called semantic primes), proposed by A. Wierzbicka  (Dixon, 1981; Wierzbicka, 1996). SP is a set of words characterized by a lack of definition. In other words, if all words from such set will be removed from the dictionary, it is guaranteed that the dictionary does not have any cycles left. Note that the SP set is not unique due to the complex structure of the dictionary.

Being able to retrieve SP set from a dictionary provides some insights on its quality: the smaller number of the primitives are included, the better, because it is easier for a reader to understand the definitions. SP sets are particularly useful for language learners as important vocabulary lists. If a person is familiar with words from the SP set, it is easier for the learners to understand the definitions, because they are familiar potentially with all concepts in any definition, even if they have to make several “steps” in the dictionary.

For future work, we can mention the idea that having SP sets for different languages, we get additional insights on the different structures of meaning, having in mind their possible comparison and analysis.

In this article, we present a method for obtaining of SP sets via genetic algorithm (GA) with single-objective constrained optimization and introduce a novel evaluation method of the obtained set. The optimization objective includes a modified version of PageRank with a cardinality constraint. Prior to fitting the algorithm, an empirical analysis is performed for the estimation of some of the parameters of the algorithms.

The evaluation method is based on the similarity of human-made word lists and the obtained set. We use WordNet dictionary for English for our experiments and manually selected from the Internet five dictionaries (word lists) for evaluation. As a result, we obtained the SP set of words, which shared similarities with some particular human-made word lists.

The article is structured as follows. In  ‘Related Work’, an overview of previous works is presented. Theoretical background concepts are discussed in ‘Theoretical Background’. Details regarding the dataset and its preprocessing are provided in ‘Data and Preprocessing’. In ‘Genetic Algorithms’, we provide background information regarding genetic algorithms. In ‘Semantic Primitives Selection with a Genetic Algorithm’, ‘Proposed Method’ and ‘Proposed Evaluation’, the proposed method and evaluation process are discussed. Further, in ‘Experiments and Results’, the final results are discussed. Finally, conclusions are drawn in ‘Conclusion’. The code is available at GitHub (https://github.com/YevhenKost/SemPrimsDetectionGA) (https://doi.org/10.5281/zenodo.7452984).

Related Work

In previous works, the concept of SP was treated both from the linguistic point of view and from the computational perspective.

The Natural Semantic Metalanguage (NSM) (Wierzbicka, 1972) is a linguistic theory, which proposes that every concept can be represented using a set of atomic terms (semantic primitives). These semantics primitives have an indefinable meaning and all natural languages have a metalanguage of semantic primitives, which serves to describe the rest of the concepts. It was estimated that there are 63 semantic primitives, which can be divided into 16 categories. Note that these primitives are too general and express more ideas about relations between objects or actions than describe objects or actions themselves.

Taking up the concept of NSM in Apresjan (1992), the use of a restricted vocabulary for the development of lexicon was proposed and it was stated that it cannot be as small as mentioned in Wierzbicka (1972). Based on this proposal, developers of the Longman dictionary of contemporary English(LDOCE) (Procter, 1978) created all definitions in their dictionary using exclusively a vocabulary with approximately 3,000 words in its latest version (which is called defining vocabulary). It is interesting to mention that in our research, we generated various sets of semantic primitives to verify its possible size. We obtained the number of elements around 2,500–3,000 elements in sets.

In Goddard & Wierzbicka (1994), the set of papers was presented for the data in various languages. In those papers, a hypothetical set of semantic and lexical universals across diverse languages were investigated empirically. According to the authors, the goal of identifying the universal human concepts is necessary since these concepts provide the basis of the “psychic unity of mankind”. The book contains works on languages like Japanese, Chinese, Thai, Ewe, among others.

In 2001, Natural Language Processing (NLP) techniques were implemented in order to identify automatically a set of words that could be considered semantic primitives. In Sidorov & Gelbukh (2001), the method, which permits building a set of candidates to be considered semantic primitives using a standard explanatory dictionary, was suggested. The authors proposed to evaluate the frequencies of the words that are reachable in a semantic network (a knowledge structure that describes the way concepts are related and connected to each other) and performed a comparison of words using a synonym dictionary and a simple derivational morphology procedure. The method implements word sense disambiguation techniques using the improved Lesk algorithm (Banerjee & Pedersen, 2002).

In Rivera-Loza (2003), Pichardo-Lagunas et al. (2014) and Pichardo-Lagunas et al. (2017), the automatic identification of defining vocabulary for Spanish using the Anaya and Royal Spanish Academy dictionaries was proposed. These dictionaries were represented as a directed graph, where each node represented a word. The graph was created by inserting word by word avoiding the existence of cycles in definitions. For each iteration, if a word closed a cycle, then it was considered a semantic primitive. In Rivera-Loza (2003), the entry words’ order was given by two methods: randomly and by frequencies with random voting. The evolutionary algorithms were used to minimize the cardinality of a selected set of words.

However, cycle detection is a time-consuming task. Performing it for every added vertex takes a lot of resources. Moreover, the order of adding vertices matters: it is possible for two identical sets to be considered SP sets in some order, but not SP sets in another one. In our work, we detected cycles only once for the selected set and the order did not matter. Our method is based on single-objective optimization with constraints, but in the previous work, the multi-objective optimization approach was used.

In Torrens-Urrutia, Novák & Jiménez-López (2022), the paradigm of Wierzbicka (1996) was changed since the authors consider that there is no difference between prime descriptors (small-tall) and prime evaluators (bad-good). In the paper, formal characterization of the concepts of semantic prime among other concepts was defined and the relations between the semantic constraints of evaluations and their sentiment were established. The authors consider semantic primes as triplets, such as (“ugly”, “medium”, “beautiful”). The primes have a prototypical (non-context) dependent meaning. In our work, we define semantic primes from a dictionary perspective, based on the relations between word definitions at word level. By our assumption, the dictionary does not provide an additional context but rather links a word with a set of other words required to understand its prototypical meaning.

For our experiments, WordNet (Miller, 1998) dictionary was selected. WordNet is a publicly available and well-known lexical database of semantic relations between words. In the database, words are linked by semantic relations, such as synonyms, hyponyms, etc. For each word in the WordNet vocabulary, short definitions are provided as well as usage examples for some definitions. However, WordNet dictionary has a set of limitations. For instance, there is no separation between the atomic and non-atomic lexical units, so it is difficult to separate multi-word constructions in the structures like collocations, idioms, proverbs, etc. In our research, we ignore non-atomic lexical units, as our algorithm is designed to work with simple words. In future work, we plan to extend the algorithm on multi-word units.

In WordNet, only words of the specific part of speech are presented: nouns, adjectives, verbs and adverbs (with some exceptions), following 1980′s generative linguistics doctrine. However, Chomsky (2002) mentioned that parts-of-speech could be distinguished with respect to their Verb-ness or Noun-ness. In the context of our research, we used a more practical approach: we considered parts of speech with lexical meaning, i.e., nouns, verbs, adjectives, and adverbs.

Finally, in the dictionary, the words with similar concepts behind them are not linked together. For example, the words “racquet”, “ball”, “net” is not related with the concept “court game”. Unfortunately, there is a similar situation in many dictionaries, and we have to use the existing linguistic dictionary information. In fact, it is word embeddings that intend to solve this problem, when the context information permits to relate semantically unrelated words.

Theoretical Background

Dictionary representation via graph

A directed graph G is defined as a tuple G = (V, E), V ≠ ∅, where V is a set of vertices and E⊆V × V is a set of edges. Each edge e ∈ E has a direction. It means that if there is a connection (v₁, v₂) ∈ E, it does not imply the connection (v₂, v₁).

In the context of this research, we define a dictionary as a resource, which for each word in its vocabulary provides a set of its meanings (definitions). The dictionary can be naturally represented as a directed graph. For every word in the definition (w_d) an edge from the defined word w_o (w_o, w_d) is added to the graph. For example, consider a word bank, which has several definitions:

• financial institution that accepts deposits and channels the money into lending activities;

• sloping land (especially the slope beside a body of water).

The edges (bank, financial), (bank, institution), …, (bank, water) are added to the graph (see Fig. 1).

The same process is applied to every word in the dictionary until the final graph representation is obtained.

Generation of permutation-based semantic primitives set

As it was mentioned before, the set of SP is not unique. In order to analyze SP sets properties, the following procedure was used.

The idea of the approach is described as follows: the word is considered to be an SP candidate if by adding it to the existing graph without cycles the cycle is created (see example in Fig. 2). In the figure, consider the word “bee” with the definition: “an insect that produces honey”. It is added to the graph. In the current graph, there are no cycles. Consider the word “honey” with the definition “a substance produced by bees”. It is added to the existing graph. Now, there is a cycle, so the word “honey” is considered to be an SP candidate. Considering each word as a vertex in the dictionary graph, they are added one by one to the graph.

More formally, if adding the vertex creates a cycle: the vertex is considered a part of the SP set and is not added to the graph. Otherwise, it is included in the graph. The order in which vertices are added is important, as with the different ordering the different SP sets are constructed. After processing of all the words from the dictionary in this way, the final SP set is obtained.

Data and its Preprocessing

For the experiments, we used WordNet dictionary for English (Fellbaum, 1998). For each word, there are different definitions with different meanings. The version of WordNet that we used consists of 86,498 defined words. We used the following preprocessing steps:

• Each word was lemmatized using Lemmatizer from the Stanza Python library (Qi et al., 2020).

• Every definition was tokenized and each token was lemmatized using Stanza as well.

• The stop words, punctuation symbols and words out of the WordNet vocabulary were removed. We used the list of stop words from the following Python library (https://github.com/Alir3z4/python-stop-words).

• The definitions, which included the word, which they are supposed to define, were removed. Others definitions for the word are kept. In this way, the self-cycles were removed during the building of a graph.

After that, the directed graph of the dictionary was built. The total number of vertices equals 86,497.

Genetic Algorithms

An algorithm based on genetic principles and natural selection is known as a genetic algorithm (GA). It mimics the evolution process and incrementally, iteratively improves the provided target objective. The “strongest” solutions “survive” the process, whereas the “weakest” ones do not.

The GA is characterized by the following concepts:

1. Initial Population or Sampling. The initial population of candidate solutions is usually generated randomly across the search space. However, domain-specific knowledge or other information can be easily incorporated. The size of the population (how many elements are considered at one iteration) is an important parameter that should be predefined.

2. Evaluation. Once the population is initialized or an offspring population is created, the fitness values of the candidate solutions are evaluated.

3. Survival. It is often the core of the genetic algorithm used. For a simple single-objective genetic algorithm, the individuals are sorted by their fitness, and survival of the fittest is applied.

4. Selection. Selection imposes the survival of the fittest process on the candidate solutions by generating more copies of those solutions with higher fitness values. The main goal of selection is to enforce better solutions over worse ones. Numerous selection techniques, such as ranking selection, tournament selection, stochastic universal selection, etc have been proposed to achieve this goal.

5. Crossover. Crossover is a process in which two chromosomes from the current generation (parent chromosomes) engage in a procedure in which some genes from one chromosome are interchanged with genes from the corresponding positions in the other.

6. Mutation. Mutation involves selection, on a random basis, of a certain number of the genes in the current population and random alterations are then made to their values. This provides a random element within the GA search process so that more search space is considered.

The GA fitting process is presented in Fig. 3.

Semantic Primitives Selection with a Genetic Algorithm

In our research, semantic primitives detection is defined as a binary subset selection problem. The problem is formulated as follows. Consider a dictionary of a total number of words N and a vocabulary set V = {a₁, a₂, …., a_N}, where a_i is a word. The dictionary is represented as a directed graph, where each word is its vertex (as it was described in previous sections). The set of semantic primitives can be represented as a binary vector of shape N, where for each word 1 or 0 is assigned: 1 if the word is considered to be a semantic primitive and 0 otherwise. So, the population is considered to be a set of binary vectors that represent the candidates to be semantic primitives.

We proposed the custom initial sampling, mutation, crossover, and fitness function for the problem.

For the initial sampling, the following procedure was applied. Consider M_p is a number of elements in the population. Firstly, we generated sets of semantic primitives as it was described in ‘Generation of Permutation-based Semantic Primitives Set’, obtaining M sets. The M was selected so that M ≥ M_p. Finally, M_p sets were selected randomly as the initial population. It provides a warm start for the algorithm and decreases the convergence speed. However, there is no guarantee that the GA would eventually find new subsets. The algorithm explores the feature space, starting from pre-generated subsets, but due to the complex nature of the dictionary graph and its cycles, the overall number of unique sets of semantic primitives can be small.

As the mutation, a random number of 0 values in binary population vector were converted to 1 (0-to-1 mutation) and 1 values were converted to 0 values (1-to-0 mutation). The number of changed values is sampled randomly for 0-to-1 and 1-to-0 mutations independently. This number was drawn from the range of predefined minimum and maximum limits.

As the crossover, we consider two parent populations P₁ and P₂ as binary vectors. The set of indexes, where both of the parents are 1, is defined as I = {i|P₁[i] = 1&P₂[i] = 1}. The output child from both of the parents is filled with value 1 in the places for the indexes from I and with random values from 1 and 0 for the remaining indexes.

We used the following fitness function: ${\displaystyle f\left(S\right)=\left\{\begin{array}{c}1,\text{if there are cycles in graph without vertices from}S,\hfill \\ -exp\left(\frac{1}{\left|S\right|}\sum _{s\in S}\text{PageRank}\left(s\right)\right),\text{otherwise},\hfill \\ \hfill \end{array}\right.}$ where S is a candidate set of semantic primitives, and PageRank is a fitted PageRank model. In order to avoid the round-off error during computation, we used an exponential function. We averaged the PageRank values in order to scale sets of different lengths. If there are cycles left, the value 1 is returned. We use this value to make the function significantly bigger, so the model is encouraged to minimize it and to avoid selecting inappropriate sets.

Proposed Method

The research approach consists of the following steps:

1. Dictionary data preprocessing and graph construction (as it was described in ‘Data and Preprocessing’).

2. PageRank algorithm fitting.

3. Generation of SP sets (as it was described in ‘Generation of Permutation-based Semantic Primitives Set’) and their empirical analysis (it is described in more detail in ‘Experiments and Results’).

4. Genetic algorithm fitting (see ‘Semantic Primitives Selection with a Genetic Algorithm’) and optimization objective that was introduced in ‘Proposed fitness function’.

Proposed Evaluation

We gathered different word lists (sets of words) from the various sources, which were manually gathered and publicly accessible. We evaluated the results on these word lists. The lists were selected as the most similar ones to the Semantic Primitives set.

For example, we used the defining vocabulary of dictionaries (gathered from common English frequency lists or by professional linguists), core vocabulary for English learners of different levels etc. The following lists were selected:

LDOCE

The Longman Dictionary of Contemporary English (https://www.pu-kumamoto.ac.jp/users_site/rlavin/resources/wordlists/LDV.html) (LDOCE) is an advanced English learner’s dictionary. The definitions are provided with a restricted vocabulary of around 3,000 words. It is widely used in Computational Linguistic research. In our experiments we used its defining vocabulary word-list.

The Britannica Dictionary (Learners’ Dictionary)

In The Britannica Dictionary (https://learnersdictionary.com/3000-words/), the editors selected 3,000 English words, which are the most important to know for students.

Longman Communication 3,000

The Longman Communication 3,000 (https://github.com/healthypackrat/longman-communication-3000) is a word list of 3,000 words. The words are selected based on the statistical properties, their usage in both spoken and written English from more than 390 million words from the Longman Corpus Network (http://www.pearsonlongman.com/dictionaries/corpus/).

Oxford Learner’s Dictionaries

The Oxford Learner’s Dictionaries (https://www.oxfordlearnersdictionaries.com/us/wordlists/) is a learner’s resource that provides 2 word lists: of 3,000 and 5,000 words. Due to the editors, those words are most important to learn for a student to speak English. 3,000 word list is considered to be a core vocabulary list while 5,000 word list is an advanced one. In our experiments, we used a 5,000 word list.

Wiktionary

Wiktionary (https://en.wiktionary.org/wiki/Appendix:Basic_English_word_list) is a free collaborative project to create a free-content multilingual dictionary. The main goal of the project is to define all words of all languages via definitions in English. In our experiments, we used 850 word list. This word list is considered to be a part of the Basic English core vocabulary. These words are commonly used in everyday life and conversations.

The idea of the evaluation method is to find the most similar word list or word lists to the predicted set. It is not expected for the predicted set to be similar to all word lists, but to one specific. The obtained word lists are different (more details are provided further, in ‘Evaluation Metric’). Finally, if the metric shows similarity at least to one word list, it means the predicted set is similar to at least one human-gathered set.

Experiments and Results

As it was described in ‘Data and Preprocessing’ and ‘Proposed Method’, in our experiments we used WordNet dictionary for English. The dictionary was preprocessed and represented as a directed graph without self-cycles. Before fitting the Genetic Algorithm (GA), we generated 1,000 different sets of Semantic Primitives (as it was discussed in ‘Generation of Permutation-based Semantic Primitives Set’). The histogram of length distribution is presented in Fig. 4.

The average length of generated SP sets equals 2,892.359 and the standard deviation equals 31.059. Based on this the SP sets tend to have similar lengths. On the other hand, how different are the generated sets? We calculated the distribution of their difference element-wise (for each unique pair the number of different elements was calculated). The distribution of element-wise difference is presented in Fig. 5. The average number is 1,479.09 and the standard deviation equals 36.48. So, based on the length and element-wise difference statistics, the generated sets are considered to be different pair-wise.

We fitted PageRank model using the scikit-network implementation (Bonald et al., 2020) with the following parameters:

• Damping factor: 0.85;

• Solver: piteration;

• Numbed of iterations: 10;

• Tolerance: 1e−6.

The fitness function distribution for the generated sets is presented in Fig. 6.

We can observe two separate clusters in Fig. 6. This fact will be investigated in our future works.

We also checked the correlation between the fitness function values and lengths of the generated SP sets. We used different correlation coefficients (Spearman, Kendall, and Pearson), but none of them showed a significant dependence (see Table 5). Two set of values are considered to be related if absolute values of correlation coefficients is around 1. The values around 0 indicate that the sets of values are independent. From the obtained results we can conclude that the fitness function values and lengths are not dependent.

In order to obtain the SP set, we run GA with the custom sampling, crossover and mutation (see ‘Proposed sampling’, ‘Proposed crossover’, ‘Proposed mutation’). The optimization objectives that we used is described as follows: (5)${\displaystyle \underset{s}{min}f\left(s\right)\text{s.t.}{\left(\left|s\right|-\mu \right)}^2\le {\sigma }^2{s}_i\in \left\{0,1\right\},i\in \left\{1,2,\dots ,N\right\},}$ where s is a Boolean vector representing the current population (SP set candidate, see ‘Proposed optimization problem and population type’), N is a number of vertices in the graph. If s_i equals 1, then the vertex i is considered to be a part of the SP set. The fitness function is then calculated as it is discussed in ‘Proposed fitness function’. The parameters µand σ are estimated from the generated sets and equal 2800 and 50 respectively. This makes constraint flexible enough for GA to output SP sets of length in range of 2,750 and 2,850. For every evaluation step GA takes for the population, the graph was built without the vertices that were considered a part of the SP set. After that, based on whether the graph contained cycles, the fitness function value was calculated and returned.

For the mutation we used a minimum number of vertices to mutate equals 0 and maximum number of vertices to mutate 60. For the population size we used 300 elements per population. We used pymoo (Blank & Deb, 2020) implementation for the experiments. We run the algorithm with 10 iterations and obtained an SP set with 2,847 vertices and the fitness function value −1.00024197 (which guaranteed that the graph is not containing cycles after the removal of the set vertices).

We evaluated the obtained SP set via the evaluation procedure (see ‘Evaluation Metric’). The results are presented in Fig. 7. From Fig. 7, Longman and LDOCE word lists showed the highest similarity to the obtained set with the BERT embeddings. So, the proposed method constructed a set of semantic primitives, which is the closest the one of the manually gathered word list and fits the SP definition.

Conclusion

In the article, we described the optimization method based on a genetic algorithm for semantic primitives detection. In this field, evaluation is always a critical problem, which is difficult to handle. We proposed an evaluation procedure based on the comparison of various word lists as targets. We fitted the algorithm on the WordNet dictionary for English and evaluated it on different manually collected word lists from various sources. Our experiments showed that the algorithm generated sets of semantic primitives that are similar to the target word lists.

It means that the algorithm is capable of generating word lists that are similar to manually compiled ones based on custom similarity measure. Specifically, for each obtained word list and target word list, for every word in these word lists, various embeddings were used. Then the cosine distance matrix was built between the obtained word list and the target word list. Hungarian algorithm was used to find the best match of words between these two word lists with some modifications to handle different lengths of the lists.

The work is a practical application of the theory of Anna Wierzbicka. We studied the reality of semantic primes, but in a specific sense related to computational linguistics. The proposed algorithm is working similarly to human judgments as was shown by comparing its results with human-generated lists of words. We showed that this task is robust to various embeddings, as we obtained similar results with various embeddings models (which means that the algorithm is consistent).

The proposed methodology can potentially be used in many linguistic fields. For instance, it can be useful for lexicographers to compare different dictionaries with respect to the number of semantic primitives in them: the less this number is, the better dictionary is. Having a smaller number of semantic primitives in the dictionary makes it easier for a human reader to understand, as it shortens the amount of required vocabulary. Based on the comparison and analysis of different dictionaries, the better dictionaries can be created based on the suggested list of semantic primitives.

Comparison of languages is another important application of the discussed methodology. As some languages (for example, Ukrainian, Polish, and Slovak) have high lexical similarity, it is expected for them to have similar sets of semantic primitives. The opposite can be assumed about languages like Chinese and Spanish: they do not share a lot of lexical similarities, so they are expected to have different sets of semantic primitives.

Furthermore, each language is changing over time. Old words and concepts are vanishing and new ones are appearing. From this point of view, it is important to keep track of changes in semantic primes over time. It may be an important indication of the transformation of the language.