Word embedding for social sciences: an interdisciplinary survey

Introduction

The advancement of machine learning technologies has revolutionized traditional research methodologies in the social sciences, enabling the extraction of insights from unconventional data sources¹ . Recently, we have been witnessing the expansion of social science research that explores textual data, captivated by the concept of “Text as Data” (Gentzkow, Kelly & Taddy, 2019; Benoit, 2020; Grimmer & Stewart, 2013). This trend is evident not only in fields that traditionally rely on quantitative analysis, such as economics! (Cong, Liang & Zhang, 2018) and finance (Rahimikia, Zohren & Poon, 2021), but also in disciplines with a penchant for qualitative analysis, like history (Wevers & Koolen, 2020). The flourishing body of literature in this domain enables social scientists to meticulously select or develop machine learning models for text mining that meet the demands of their research, providing a vast array of options. While rule-based models, such as dictionary-based models, remain popular (Tausczik & Pennebaker, 2010; Hutto & Gilbert, 2014), neural network-based embedding models are gaining increasing traction in social science research.

Embedding models learn low-dimensional representations of entities from data by predicting relationships between these entities (Boykis, 2024). The advantage of embedding models in social science research lies in their ability to acquire the “embedded” structure in an unsupervised manner. This inherent flexibility allows social scientists to apply these models to their data with minimal constraints, facilitating their analyses. These entities could be nodes in a graph (Goyal & Ferrara, 2018; Barros et al., 2021; Cai, Zheng & Chang, 2018), pixels in an image (Liu et al., 2020; Baltrušaitis, Ahuja & Morency, 2018), or vertices in a network (Cui et al., 2018; Zhou et al., 2022a). Despite the multitude of embedding algorithms available for diverse purposes, word embedding models have been recognized as some of the most widely adopted methods (Chaubard et al., 2019; Mikolov et al., 2013a, 2013b; Goldberg & Levy, 2014; Mikolov, Yih & Zweig, 2013; Mikolov et al., 2015). Among these, the most accessible word embedding model for social science research is the word2vec model (Mikolov et al., 2013b; Mikolov, Yih & Zweig, 2013; Mikolov et al., 2013a), used in diverse fields ranging from management science (Lee & Kim, 2021; Toubia, Berger & Eliashberg, 2021; Chanen, 2016; Li et al., 2021; Bhatia et al., 2021; Chen & Zhang, 2023; Zheng, 2020; Khaouja et al., 2019; Tshitoyan et al., 2019) to poetry (Nelson, 2021).

The rise in the use of word2vec and other related models in social science research has not been accompanied by adequate documentation of how these models are utilized. However, because this trend spans multiple disciplines, we are witnessing a lack of sufficient cross-communication. First, this absence of an integrated knowledge framework hampers discussions about the broader impact and relevance of word embedding in social science research. Second, the absence of comprehensive surveys spanning multiple disciplines has led social scientists to reference only literature within their own fields, inadvertently leading to the reinvention of methodologies already developed in other domains. Consequently, this solid approach often results in deserving scholarly works not receiving the recognition and citation they merit, thus undermining their rightful academic evaluation. Therefore, we need to review the broad spectrum of studies that should be referenced, many of which fall outside their immediate expertise.

To address the knowledge gap resulting from a lack of connections between disciplines, this article aims to identify emerging interdisciplinary trends in machine learning models arising across diverse fields. In this literature review, we seek to bridge the knowledge gaps between different fields and ask the reseach question “How similarly are word embedding models, such as word2vec, applied across different methodologies in social science research?” This literature review builds a taxonomy of these application methodologies to illustrate how social scientists harness word embedding models, particularly focusing on word2vec. The taxonomy emphasizes the applications of word embedding published in social science journals, outlining the contemporary landscape of social science research utilizing word embedding models, rather than delving into the technical intricacies of these neural network models. This literature review targets social scientists who are currently using or considering the use of word embedding models like word2vec in their research. Therefore, the presented article does not comprehensively review the state of the art in the technical aspects of embedding models. Instead, we offer a review that covers various social science disciplines, especially for those who may not specialize in machine learning or natural language processing. To provide an overview of such different fields, we develop a taxonomy to guide potential readers in situating their studies within this interdisciplinary literature.

This interdisciplinary survey covers literature published across a wide range of research fields, helping researchers or practitioners access studies and insights outside their own areas of expertise. In addition, the taxonomy developed in this survey helps researchers in positioning their work within the context of the literature, which is too broad to be referred in a article in a single field. In this way, the interdisciplinary literature review offered here contributes to bridge barriers among research fields. This survey also helps us understand and enhance the application of word embedding models in social science contexts.

The structure of this survey is as follows. After the discussion of search methodology, we begin this survey with a brief overview of the technical background of word2vec in “Word Embedding Models”. Then, we introduce the nine labels used in our taxonomy (Table 1) and review the articles detailing the taxonomy throughout “Pre-trained Models” to “Non-text”. We discuss a pitfall of similarity measurements with a simple experiment. In addition, we employ the proposed taxonomy to understand the literature using the word2vec model for social science.

Survey methodology

This survey primarily focuses on articles published in social science journals, aiming to conduct a concentrated literature survey. Although our main attention was not directed toward research in word embedding published in computer science conference proceedings, we selectively included some of these articles to ensure a comprehensive understanding of the field. This approach was essential to paint a complete picture of the literature landscape.

We primarily focused on articles that had a clear application or discussion of word embedding models in a social science context. We gave preference to articles published in peer-reviewed journals and conference proceedings to guarantee the credibility and academic rigor of our sources. Conversely, we excluded articles that did not fall within our specified subject areas. Non-English publications were also excluded due to language constraints. This exclusion was necessary to ensure that our sources were reproducible and verifiable by our readers.

For this survey, we predominantly used the Scopus Database, implementing a specific query that included terms word2vec and combinations such as word AND embedding. To ensure that our survey was as exhaustive as possible, we expanded our search to include Google Scholar and the Social Science Research Network (SSRN) to include working articles or preprints, adapting our query to suit their unique search algorithms. This process is not limited to research articles but also includes other types of publications, such as articles, book chapters, and review articles. Additionally, we compile articles that cite (Mikolov et al., 2013b; Mikolov, Yih & Zweig, 2013) to capture the literature on the application of word2vec. From the collected articles, we select those published in fields related to social sciences. The specific fields we cover are listed in the research area column in Tables 2 and 3.

Through this meticulous and methodical approach, we aimed to ensure a comprehensive and unbiased coverage of the literature. This strategy was crucial in facilitating a robust analysis of the current state of word embedding applications in social science research, thereby contributing significantly to our understanding of the field. This survey focuses the foundational role of the word2vec model introduced by Mikolov et al. (2013a) and we review the literature from 2013 to the present (2024). In addition to the word2vec application, we also acknowledges earlier research, including the seminal work of Rubenstein & Goodenough (1965) from 1965.

Word embedding models

We briefly review the technical aspects of word embedding models before diving into the main part of our survey. Since our focus is the literature that uses word2vec models, this section discusses the word2vec algorithm. This section ensures that the presented article provides sufficient technical prerequisites of word2vec, while in-depth explanations are relegated to relevant literature sources.²

Word embedding models learn the relationships between words in sentences in the data and return embedding vectors based on their learned model. The embedding vectors are considered to represent the semantics of the words in the sentence, and the distributional hypothesis Harris (1954) supports this assumption. The theory asserts that a word is determined by the surrounding words, and experiments with human subjects have validated that word embedding models can capture semantics similar to human cognition (Caliskan, Bryson & Narayanan, 2017). This prompts social scientists to assume that studying word embedding models learned from the text of interest can reveal how humans process information embedded in the text.

Evaluations of word embedding models roughly take two forms: intrinsic evaluation and extrinsic evaluation. Typical tasks for intrinsic evaluations are analogy and similarity. Many datasets for intrinsic evaluation are publicly available and accessible (Analogy (Miller & Charles, 1991; Charles, 2000; Agirre et al., 2009; Rubenstein & Goodenough, 1965; Bruni, Tran & Baroni, 2014; Radinsky et al., 2011; Huang et al., 2012; Luong, Socher & Manning, 2013; Hill, Reichart & Korhonen, 2015), Similarity (Mikolov et al., 2013a; Mikolov, Yih & Zweig, 2013)). For extrinsic evaluation, they often use word embedding models for translation or sentiment analysis. See Wang et al. (2019) for recent evaluation methods.

Although there are various word embedding models, social scientists have favored word2vec as the most popular model due to its simplicity, high user-friendliness, and longstanding usage (Chaubard et al., 2019; Rong, 2014). Thus, this survey article primarily focuses on the application of word2vec in social science research while also covering other language models that describe the new trend of this interdisciplinary research.

Taxonomy of applied methods and labeling literature

Now, we turn to the main focus of this survey article, which provides an overview of a new emerging trend that applies word embedding models to social science research. To give a clear overview of the applications of word embeddings in interdisciplinary research, we first develop a taxonomy of the analytical methods used in the surveyed social science articles. We then label the literature based on the developed taxonomy. This approach allows us to review the studies from a concrete perspective on the trend of word embedding applications, which spans a wide range of disciplines.

We first review the literature on applications of word embedding models for social science. Our taxonomy introduces the labels of popular methods found in the surveyed literature. This method-oriented classification benefits us when surveying interdisciplinary topics. The nature of these topics prevents us from discussing each article according to its research topics or questions because showcasing such highly diverse outcomes dilutes the visibility of the survey. Therefore, instead of following the literature of each research field, we study the way the surveyed articles use word embedding models for their analyses.

We discuss our taxonomy and labels with the literature in “Taxonomy”, and we summarize the labels in Table 1. We define eight labels and summarize the descriptive definitions of the analytical methods. We also list the surveyed articles in Tables 2 and 3 with the corresponding labels. This process identifies several important research orientations in the literature. For example, we identify a popular method that constructs variables for study using reference words (W/reference label). In this survey, we also observe a new line of literature that applies word2vec to non-textual data, such as relationships between users on web platforms.

Cosine similarity or euclidean distance?

We find that many articles examine the similarity between embedding vectors when they employ the methods discussed so far such as reference words (“Reference Words”) or non-text (“Non-text”). For similarity measurement, most use a cosine similarity but Euclidean distance ( $L_2$ norm) is also popular. Because the similarity between vectors can be interpreted as the distance between vectors, we have various options to define the distance between vectors⁷ . However, it is not trivial whether the results can change depending on the measure we use for analysis. Some articles analyze the robustness of the results using multiple options, such as Euclidean distance and cosine similarity Garg et al. (2018).⁸

Toubia, Berger & Eliashberg (2021), in their supplementary information, argues that Euclidean distance is richer than cosine similarity that only measures the angle. Indeed, cosine similarity is not a perfect alternative to Euclidean distance. Cosine similarity can disregard the information on the distance between two given vectors. Figure 1 depicts five different vectors. Vectors 1 and 2 have the same angle ${\theta }_1$, and therefore, the cosine similarity of these is one. In contrast, the Euclidean distance between the two is $d$, which means they are different in terms of Euclidean distance.

It is important to recognize that equal Euclidean distances do not always correspond to equivalent cosine similarity values and that Euclidean distance cannot always be interchanged with cosine similarity, nor vice versa. The Euclidean distance between Vectors 2 and 3 is the same as the Euclidean distance between Vectors 1 and 2, but these two pairs have different angles ( ${\theta }_1$ and ${\theta }_2$). Figure 1 also depicts that using the normalized Euclidean distance eliminates the distance between Vectors 1 and 2. The normalized Vector 1 will be on the same point as Vector 2, which is on the unit circle. In this case, these two vectors are the same from the perspective of both cosine similarity and Euclidean distance. Also, we should note that small distances do not always mean small cosine distances. For example, Fig. 1 depicts Vectors 4 and 5 is small in Euclidean distance compared to the other vectors, but their angle is ${\theta }_1$ (approximately $\pi /4$); therefore, the cosine similarity between them is not small.

Specifically, the relationship between two measurements becomes dismissed when $\theta$ is small or around $\pi$. Let us consider the distance between two points in a unit circle: A $(0,1)$ and B $(\cos \theta ,\sin \theta )$.We can consider two points in a unit circle by $(x,y)$ and $(x\cos \theta -y\sin \theta ,x\sin \theta +y\cos \theta )$, where the angle between the two is $\theta$. Since we only study the distance between the two, we set $x=1$ and $y=0$ without loss of generality. The distance between two is $(2(1-\cos \theta ){)}^{1/2}$, and the cosine similarity is $\cos \theta$. Therefore, the error between the Euclidean distance of normalized vectors and cosine similarity is a function of $\theta$, and that is

(2) $$e(\theta )=\cos \theta -(2(1-\cos \theta ){)}^{1/2}.$$

Since the first derivative of $e(\theta )$ with respect to $\theta$ is

(3) $$e\mathrm{\prime }(\theta )=-\sin (\theta )\left(1+(2{(1-\cos (\theta ))}^{1/2})\right),$$the relationship between the two metrics become zero around ( $\theta =\pi$ or $0$) where angle $\theta \in R$. This fact implies that the two metrics are incompatible when their angle is small or large. We also note that the fact that the angle is not always relevant to the distance.

To further discuss the comparison between the cosine similarity and Euclidean distance, we study the empirical relationship between them using the Google News pre-trained word2vec model that contains the embedding vectors of 3,000,000 words Google (2013). We construct 1,500,000 random word pairs and plot the relationships in Figs. 2A and 2B. While Fig. 2A demonstrates that the two metrics are correlated, their correlation is not strong (the Pearson correlation coefficient $\rho =-0.357$), which implies that using cosine similarity would lose some information and vice versa. Figure 2A also demonstrates that the cosine similarity distributes uniformly where the Euclidean distance is small, meaning that the two different pairs of vectors with the same Euclidean distance can assume different values in their cosine similarities. This deviation becomes large where the Euclidean distance is small. This finding is consistent even when we normalize the metrics. Figure 2B demonstrates the empirical relationship between the Euclidean distance of normalized vectors and cosine similarity. While they capture the difference of two word embedding vectors in a similar way (the Pearson correlation coefficient $\rho =-0.991$), they are not the same. The Euclidean distance of normalized vectors is more sensitive than cosine similarity in which the distance between two points is close (i.e., $\theta$ is small) or large (i.e., $\theta$ is around $\pi$).

The above discussion suggests that the cosine similarity and Euclidean distance can qualitatively return in a similar manner, but they are not always compatible. Notably, the cosine similarity can capture the difference between two vectors when their Euclidean distance is small. However, when we study the normalized vectors, the Euclidean distance captures the differences, but the cosine similarity does not when the Euclidean distance is small. Given this, we should know what characteristics we intend to capture when selecting a metric.

These findings also imply that the Euclidean distance and cosine similarity (angle) capture different characteristics. Recently, some studies in computer science literature have proposed word embedding models that explicitly model this relation, such as hierarchical relationships (Tifrea, Bécigneul & Ganea, 2018, Nickel & Kiela, 2017; Vendrov et al., 2016; Vilnis & McCallum, 2015). For example, (Iwamoto, Kohita & Wachi, 2021) proposed the embedding models that use the polar coordinate system and illustrate that their model captures the hierarchy of words in which the radius (distance) represents generality, and the angles represent similarity.

Conclusion

This survey comprehensively overviewed the emerging interdisciplinary research trend. We survey the literature on the application of word embedding models to social science. Since this research trend covers a wide range of research areas, we first construct the taxonomy of the method used in the surveyed articles, constructing the labels of popular methods. We then discuss the literature with the constructed labels and this helps us with documenting and bridging the literature in different research areas from neuroscience to humanity. In addition, we discuss several important issues regarding the similarity measurement, which researcher should note in their application.

We have made several contributions to the literature and the science community. First, this survey detects the emerging trend of an application in social science that uses word embedding algorithms, and we document this trend not only by focusing on a single research area or discipline. This survey would serve “a hub” for researchers from various academic fields to refer to findings from other disciplines when using word embedding models for social science analysis. The second contribution is the introduction of the taxonomy of the methods used in the surveyed article. Based on this, we provide the labels that would help researchers classify their work based on their methods. Before this work, the literature has not particularly contextualized their work based on the way they use word embedding because of the lack of notions that bridge the literature. The present research also provides researchers with useful guidance and a well-organized framework for how to explore or classify these applications of word embedding models in social sciences. Lastly, we contribute to the literature by summarizing several important issues when applying word embedding to social science research. More specifically, the survey discusses some pitfalls of using popular similarity measurements. We would like to expect that this survey article harnesses the researcher working on this interdisciplinary area to explore and contextualize their studies, connecting their work across different research areas.

While the present article contributes to the science community in the aforementioned points, it has limitations as the other works do. First, we mainly focused on the application of the word2vec algorithm and did not deeply explore ones utilizing other embedding algorithms or neural-based approaches. For example, this survey did not cover the sentiment analysis or topic modeling by machine learning including the embedding-based method such as BART (Lewis et al., 2020). In addition, the presented work did not deeply discuss the methodological discussion about word embedding. There are a substantial amount of topics about the theoretical aspect of word embedding algorithms as the famous quote goes “Good question. We don’t really know.” in Goldberg & Levy (2014). The further direction for this literature could establish the theoretical foundation and empirical evidence that provides a framework or procedure for researchers who apply word embedding algorithms to social science research.