Unsupervised query reduction for efficient yet effective news background linking

Ad-hoc based retrieval of background links

Some proposed methods addressed the task using an ad-hoc search approach, in which an input query article is analyzed to construct a search query. Yang & Lin (2018) introduced a number of methods that were all implemented in Anserini (Yang, Fang & Lin, 2017). The proposed methods selected the query terms as the 1,000 article terms having the highest TF-IDF values, or as the first 1,000 terms that appeared in either the article or some selected paragraphs. Since the average length of the articles in the Washington Post collection was around 400 terms (after removing stop words), we can say that this method, for many of the articles, used the full article as a search query. Lopez-Ubeda et al. (2018) constructed different queries using the article’s title, the full body text, and both the title and the body, to generate initial results. Candidate articles were then selected based on a k-means clustering to diversify the results to the news reader.

In an attempt to find the most influential terms to be used in the search query, Bimantara et al. (2018) selected the key phrases extracted using TextRank (Mihalcea & Tarau, 2004) (a basic graph-based text analysis algorithm), and used these keyphrases as the search query along with a list of named entities extracted also from the query article. Documents were also re-ranked considering their publication dates to allow articles that were published close to the input article to appear at the top of the retrieved list. Essam & Elsayed (2019) also used graph-based keyword extraction mechanisms to determine the most influential terms from the query article, and used up to 20% of the extracted terms from the query article as a search query.

Forming multiple search queries from the input article, Lua & Fang (2018) suggested splitting an article into paragraphs and extracting query terms from each paragraph to form the search queries using a probabilistic model. Before indexing, the entities in the text were identified using DBpedia Spotlight (Daiber et al., 2013), and they were subsequently replaced by their canonical forms (Lu & Fang, 2019). In 2019, the authors suggested, additionally, to assign different weights to the spotted DBpedia entities in the input document based on its surrounding context. To do so, they combine the words before and after each occurrence of each entity, and generate a language model based on the words. They then compute the KL-divergence between this context language model and the document language model to find the entity weight. The probabilistic weight distribution of the regular words and of the entities of the query article is then used in the retrieval process of background links.

Exploring relevance feedback and query expansion, Missaoui et al. (2019) initiated a search first using the named entities extracted from the query article’s title. They then used the top 10 retrieved articles in expansion to update the search query. Ding et al. (2019) also adopted similar approach. They did not use only named entities though for their initial query; instead, they used the title and body of the input article with weights equal to 0.3 and 0.7 respectively. They also assumed that the top 10 retrieved articles are relevant and the lower 10 retrieved documents from a 100 set are non-relevant. For retrieval, they experimented with BM25 and Query Likelihood models, and for expansion, they studied Rocchio and RM3.

Supervised retrieval of background links

To learn a model that estimates the usefulness of a background article to a query article, Foley, Montoly & Pena (2019) constructed different linear learning to rank retrieval models trained on the 2018 collection. To select the best set of features for the model, the Coordinate Ascent model (Metzler & Croft, 2007) was adopted. Similarly, Qu & Wang (2019) trained a multi-class classifier to re-rank background documents using logistic regression. The classifier classifies the candidate articles from 0 to 4 according to how much knowledge they provide to the query article. To produce the background documents, the top 80 frequent words (using TF-IDF scores) were used as a query to retrieve an initial set of 1,000 candidate background documents. This set is then re-ranked by the classifier. Khloponin & Kosseim (2019) suggested representing all documents in the collection as vectors using Doc2Vec distributed representation (Le & Mikolov, 2014). Based on the assumption that documents that have some background knowledge to the query document are closer in the vector space, they then computed the cosine similarity between the query document and all other documents in the collection. Similar to this work, Day, Worley & Allison (2020) computed embeddings for the documents using Sentence-BERT. Precisely, an initial set of background links were first retireved using the highest in TF-IDF terms as a search query, then for each document in this set, an embedding is generated by pooling the embedding vectors of the leading three paragraphs generated by Sentence-BERT. Finally, the cosine similarity between the embeddings of both the query and each of the candidate articles was used to generate the final rank of background links. Khloponin & Kosseim (2021) further explored a number of document embedding representations, with different proximity measures, and found that GPT2 and XLNet embeddings generally lead to higher performance in the semantic representation of articles for this task.

Computing their own embedding vectors for the dataset, Gautam & Mandar Mitra (2020) trained a word2vec model on all articles, where each word in the corpus was represented as a 300 dimensional vector, and an article was represented as the sum of the vectors corresponding to the top 25 words contained in the article. To retrieve the background links, the authors initially retrieved a set of articles using a search query composed of the 80 highest terms in TF-IDF values. Documents were then reranked given thier retrieval score and the cosine similarity between its embeddings and the query article embeddings. Deshmukh & Sethi (2020) also computed their own embedding representation of the query and candidate documents, by first extracting important keywords from the text, then treating the concatenation of these keywords as a sentence that is fed to Sentence-BERT to obtain the required embeddings.

Ak et al. (2020) suggested further two supervised based methods for background linking. The first used BERT-based extractive summarization methods to generate a summary of 180 words for the news article, then used this summary as a search query. The second fine-tuned next sentence prediction task in BERT for background linking. Since BERT input is limited to 512 tokens, both the query article and the candidate background article were cropped to half of 512. In an attempt to analyze the different topics discussed in the news articles and how much the common topics between articles affect relevance in this problem, Ajnadkar et al. (2021) used a trained LDA model to analyze the news articles in the Washington Post dataset and obtain its topic distributions, then find the most similar articles to a given query article based on the maximum cosine similarity with its topics.

PositionRank

PositionRank is a also a graph-based keyword extraction algorithm (Florescu & Caragea, 2017) that uses the graph-of-words structure. However, after constructing the graph, PositionRank adapts the PageRank algorithm (Page et al., 1999), a well known iterative random walk-based algorithm for ranking web pages, to assign scores to nodes (words) in the graph that indicate its importance. Aside from PageRank that assigns equal initial scores/weights to all nodes in the graph, PositionRank assigns initial scores to nodes (words in this case) based on their positions in the text being analyzed. Precisely, it favors more nodes that occur at the lead of the text, based on the assumption that keywords occur frequently very close to the beginning of a document. In constructing the graph of words, PositionRank selects only nouns and adjectives to be added as nodes to the graph. After the graph analysis, PositionRank considers noun phrases that match the regular expression (adjective)*(noun) + of length up to three to create unigrams, bigrams, and trigrams. It assigns bigrams and trigrams the sum of the scores of its individual unigrams. To create the news background linking search queries using PositionRank, and since we only care about weighting single terms, we skip this process of creating bigrams and trigrams, and use the weights assigned by PositionRank to the unigrams as is.

TopicRank

TopicRank is another graph-based keyword extraction algorithm (Bougouin, Boudin & Daille, 2013) that is also based, as PositionRank, on the idea of the PageRank algorithm. However, in TopicRank, the graph nodes are not only single words; instead, a document is represented as a complete graph of topics and the relations between those topics. A topic is defined as a cluster of similar single and multi-word phrases. To define these phrases, TopicRank uses a part-of-speech (POS) tagger and extracts the longest sequences of nouns and adjectives from the document as keyphrase candidates. Keyphrase candidates are then split into clusters using a Hierarchical Agglomerative Clustering (HAC) algorithm. After creating the different topics and representing them as nodes, a complete graph is created, where edges are added between topics given their closeness to each other. The weight on the edge between topics is calculated using the reciprocal distances between the offset positions of their keyphrases. The basic PageRank on this graph is then applied to weigh the different topics. Finally, TopicRank selects the keyphrase from each topic that appears first in the document, and assigns it the topic’s weight. For constructing the search query, we split the extracted keyphrases into unigrams, then we compute the score or boost value of a unigram term u as follows: $score(u)=su{m}_{i=1}^{k}TR(i),$ where k is the number of the extracted keyphrases that contain the unigram u, and $TR(i)$ is the weight assigned by TopicRank to an n-gram keyphrase i.

Multipartite Rank

Based on the idea of creating better representation of keyphrases within the different topics in the a text, Boudin (2018) proposed Multipartite Rank (MPR). MPR uses a more dense graph structure than with TopicRank, called multipartite graph to model the topics with their key phrases. In that graph, the nodes are candidate keyphrases that are connected only if they belong to different topics. Accordingly, the document is represented in this graph as tightly connected sets of topic-related candidates/keyphrases. MPR also adds an adjustment on the weights of the keyphrases that occur first on each topic. This is based on the assumption that candidates keyphrases that occur at the beginning of the document are better representatives of the topics. We follow the same method as with TopicRank to select the required terms and weights.

sCAKE

Following the line of graph-based keyword extraction methods, sCake (Duari & Bhatnagar, 2019) was motivated by the need to design a paramterless graph construction method. Instead of a sliding window parameter as in k-Core or k-Truss, in sCake, the window slides over two consecutive sentences and the terms co-occurring in these sentences are linked (i.e., the edge between a node n_i and node n_j holds how many times the terms t_i and t_j occurred in two consecutive sentences). Similar to PositionRank, sCake only considers nouns and adjectives as candidate terms for constructing the graph of words. To score nodes in the graph for keyword extraction, sCake initially applies the k-Truss method explained above, then uses the created truss-hierarchy of subgraphs to assign different features to the nodes, and aggregates those features for each node to compute its final score. These features are the maximum truss number at which an edge between the node and any of its neighbors resides, the sum of truss levels of the node’s neighbors, a positional feature that attempts to favor nodes at the beginning of a document as in PositionRank, and a semantic connectivity score that measures the number of distinct concepts that a node links to, assuming that the more the node’s neighbors belong to different concepts, the more important it is. Since sCake creates only unigrams, for constructing our search queries, we use the score assigned by sCake to each output term as is.

Yake

Unlike graph-based methods, Yake is a completely statistical method that was introduced recently for keyword extraction (Campos et al., 2020). It depends on capturing five statistical features of terms in a document, and integrating those feature scores in a final score for keyphrase determination. The first feature captures how frequent a term was mentioned starting with a capital letter excluding the beginning of a sentence, or marked as an acronym. This is based on the assumption that uppercase terms are usually more relevant as keywords than lowercase ones. The second feature is term position, which captures, as in PositionRank, the position of terms in the document, favoring terms that occur at the beginning of the document. Unlike PositionRank though that captures the position of terms itself, Yake captures the position of the sentences in which the terms occur. The next feature is a normalized version of the term frequency in a document. The fourth captures the context around the term, based on the assumption that the higher the number of different terms that co-occur with the candidate term on both sides, the less significant term t will be. Finally, the last feature captures the percentage of different sentences in which a term occurs, assuming that terms which occur in different sentences are more important. In Yake, the smaller the value of a term, the more significant it is. Since in our work we assign boost weights to extracted terms, we assign a score to the term t as follows: $score(t)=1/Yake(t)$.

Dataset

We conducted our experiments on version 3 of the Washington Post news test collection released by TREC for the background linking task in the news track (https://trec.nist.gov/data/wapost/). The collection covers 7 years of articles (2012–2019) with about 672 k documents comprising news articles, columns, and blogs. It also contains 50, 57, and 49 query articles released in 2018, 2019, and 2020 respectively. Each of those query articles is associated with a set of background articles manually-judged on a 0–4 relevance scale indicating how much context and background knowledge they provide to the query article (Soboroff, Huang & Harman, 2018, 2019, 2020).

Preprocessing and indexing

Each article in the dataset is formatted as a JSON object, which is broken into a title and multiple content paragraphs. For each article, we extracted the metadata (title, author URL, and publishing date) and concatenated the text content (marked by a “sanitized_html” type). We used JSOUP library (https://jsoup.org/) to extract the raw text from the HTML text. Afterwards, we lower-cased the text and removed stop words. Finally, the pre-processed text was indexed, along with the article’s meta-data, using Lucene v8 (http://lucene.apache.org/). We did not perform stemming as, per preliminary experiments with the full-article baseline, the performance was degraded compared to non-stemming.

Baseline

Since the news background linking problem was mainly and extensively addressed within the scope of TREC conference, we elected to choose the baseline as the most (to date) effective method as reported by TREC (Soboroff, Huang & Harman, 2018, 2019, 2020). It uses the concatenated text of the query article’s title and its body content (after stop words removal) as a search query to retrieve the background links. While there were few attempts outside TREC to address this problem, none were able to achieve a better performance than this simple baseline method, constituting the state of the art on that problem. Accordingly, we focus our comparison with only this baseline, aiming at achieving the same effectiveness, but with much shorter, hence efficient, search queries.

Implementation issues

Due to lack of open source implementation, we implemented our k-Core and k-Truss keyword extraction methods in Java. For PositionRank (https://github.com/ymym3412/position-rank), Yake (https://github.com/LIAAD/yake), and sCake (https://github.com/SDuari/sCAKE-and-LAKE), we used the authors’-provided implementation developed in Python for the first two methods and R for the third. For TopicRank and MPR, we used the PKE Python library (https://github.com/boudinfl/pke). We set the sliding window to three for methods that required it (i.e., PositionRank, k-Core, and k-Truss). For Yake, we set the maximum n-gram size to one to get only unigram keywords, and the window size to 1, as recommended in Campos et al. (2020). For the other methods, we used the default parameters settings as suggested by their given libraries. For instance, for MPR, we set alpha to 1.1, and the threshold to 0.74. For PositionRank, we set alpha to 0.85.

Retrieval

After keyword extraction, i.e., generating the search query, the constructed index was used to retrieve the set of candidate background articles. We used the default Lucene scoring function, which is an implementation of the BM25 retrieval model, to score the articles in all of our experiments¹ . Upon retrieval of the candidate articles, we filtered the results to exclude articles that are “Opinions”, “Letters to the Editor”, or “The Post’s View” as they were declared by TREC to be non-relevant. We also excluded articles that were published after the query article.

Evaluation measures

We used nDCG@5 as our primary evaluation measure for effectiveness, since it was used as the primary measure by TREC. The gain value for each retrieved background article was calculated as 2^r, where r indicates the relevance level between 0 (non relevant) and 4 (the most relevant). For reliable evaluation, we evaluated the studied methods using 3-fold cross validation over the TREC query articles. For easier future comparisons, the folds were chosen to be the query sets released by TREC in each year for the background linking task (2018, 2019 and 2020), i.e., two TREC query sets of two years were used for parameter tuning, and the third was used for testing. The only parameter we tuned in our experiments was the number of extracted terms that are used to construct the search query.

To report efficiency, we use the query response time in milliseconds. We ran all our experiments on a MacBook Pro machine with a 2.5 GHz Quad-Core Intel Core i7 processor, and 16 GB 1600 MHz DDR3 memory. The reported time per query is the average time of three runs of the same experiment.

Leading paragraphs as search queries (RQ1)

When authors write news articles, they often adopt the inverted pyramid style (DeAngelo & Yegiyan, 2019; Pottker, 2003), in which the essential and most attention-grabbing elements are introduced first in the article. Accordingly, in this section we address RQ1, that is to test if the lead paragraphs of the articles can simply and sufficiently be used as a search query instead of the whole article for the retrieval of the background links. In other words, we wanted to check if applying keyword extraction is at all required.

To answer this question, we experimented with constructing search queries simply using the top paragraphs of the query article, after stop words removal. Figure 4A shows the histogram of the length of the query articles in paragraphs² . The figure indicates that most articles have less than 20 paragraphs, with an average of about 18 paragraphs. Figure 4B illustrates the performance when we vary the number of top paragraphs from 1 (just the first paragraph of the article) up to 30 (almost the full article). We observe that increasing the number of paragraphs used for constructing the search query improves the performance. We also note that the peak nDCG@5 occurred at using 16 paragraphs, which is very close to the average number of paragraphs. This indicates that using only a few lead paragraphs is not sufficient for an effective background links retrieval.

As for the efficiency of the retrieval process, Fig. 5 shows the average query response time using the lead paragraphs as search queries. As illustrated, there is a high growth in response time while increasing the number of paragraphs, which increases the number of terms in the search query³ . This experiment confirms the need for a keyword extraction mechanism that can select few terms to construct the search query while maintaining the effectiveness of using the full article.

While we might expect that the baseline method should always exhibit the highest response time, the figure shows it is not the case when using 25 to 30 paragraphs. This is due to the optimization of query processing performed by the Lucene platform, which adopts BlockMax WAND approach that skips scoring some documents in the posting lists of terms when their score contribution is expected to be not competitive (hence not affecting the document ranking), resulting in a faster retrieval process (Grand et al., 2020). This indeed is more evident in long queries, as in the baseline case.

Keyword extraction for search query reduction (RQ2)

The next experiment aims at addressing RQ2, which is concerned with measuring the effectiveness of the keyword extraction techniques in search query reduction for background linking. To conduct this experiment, we applied each keyword extraction method explained earlier to analyze the query articles and construct search queries. Figure 6 shows the cross-validation results varying the number of extracted keywords from 30 (since our preliminary experiments in this regard showed that the effectiveness degrades noticeably if the number of search terms are less than 30) to 100 (for efficiency purposes). It also reports the performance of the Baseline method. The range of the optimal (after tuning) values of the number of extracted keywords (over the three folds) for each method is reported on top of the respective bar. Surprisingly, all methods, except sCake, exhibit very similar performance with small differences in effectiveness. Accordingly, we applied the paired two-sample t-test with 5% significance level between the baseline and each of the other methods; we found that the differences in performance between the baseline and each of Yake, k-Truss, k-Core, PositionRank, and MPR methods are not statistically significant, which means that those five methods essentially exhibit the same effectiveness of the baseline despite using much shorter queries. This, in turn, achieves our goal of having no sacrifice in the retrieval effectiveness as a result of the query reduction process.

Being effectively similar, using any of the methods that exhibit no statistical significant difference with the baseline is expected to be more efficient than the baseline while maintaining the retrieval effectiveness. This is due to the much shorter search queries. Recall that the average length of the query articles is 400 terms vs maximum of 100 terms in the above experiment, i.e., 75% reduction in the query length. However the selection of the query terms might impact the speedup. To check that, we computed the average query response time for each of the keyword extraction methods for 100-term queries, as shown in Fig. 7. In this experiment, we did not tune the number of terms, as we choose to test the worst case scenario, which is using our ceiling number of terms (i.e., 100). It is then intuitive that the time taken for processing a subset of those terms will be lower. Among the five methods above that exhibit the indifferent performance with the baseline, Yake is the fastest with about 6.2× speedup over the baseline (which takes 1,645 ms, but omitted from the figure due to scale). Recall that Yake is also the closest in nDCG@5 performance to the baseline, among all other methods (as illustrated in Fig. 6), and also the fastest (after the two TF-based methods) in the keyword extraction process (as shown in the time complexity analysis earlier). It indeed yields a perfect solution for the practical scenarios. We also notice that the queries generated by the TF-IDF method are the most efficient ones. This is expected as the method prioritizes terms of high IDF, i.e., short postings lists. Therefore, if we need even higher speedup (12.4× over the baseline in this case) while tolerating some degradation in effectiveness, TF-IDF is then the choice.

The results above inspired us to check if the terms in the queries generated by Yake are different than the ones generated by the TF-IDF method (hence the difference in the processing time), and if those different terms actually influence the retrieval effectiveness. To answer this question, we computed the average cosine similarity between the search query vectors generated by both methods over queries of different lengths. The resulted average similarity was 0.8, indicating that TF-IDF yields somewhat different queries. That motivated us to conduct another exploratory followup experiment, where we only keep the terms that are commonly extracted by TF-IDF and Yake. Figure 8A shows the average length of the queries after this filtering for different number of originally extracted keywords. The figure shows considerable reduction; for example, 27 terms were filtered out on average when the original queries have 100 terms, keeping only 63 terms in common. As for the effectiveness past the reduction of the queries, it can be seen in Fig. 8B that when the originally-extracted terms exceeds 80, the reduced queries yield very close effectiveness, in terms of nDCG@5, to the original queries generated by Yake. In fact, the difference in performance between the baseline and the query reduced from 100 extracted terms is not statistically significant, indicating that the removed terms were quite ineffective, yielding a more efficient retrieval. This can be clearly noticed in Fig. 8C, which shows that the reduced queries are much more efficient than the queries generated by Yake, and a bit more efficient than the queries generated by TF-IDF, yielding about 13.3× average speedup over the baseline.

In conclusion, several keyword extraction methods exhibit similar effectiveness (with no statistical significance difference) to the baseline while being much more efficient. Among those, the queries generated by Yake, in particular, yield 6.2× speedup over the baseline. Moreover, Yake’s queries can be further reduced by filtering out terms that were not selected by TF-IDF method, yielding even better average of 13.3× speedup in response time over the baseline, with a little cost of running both (already fast) methods of keyword extraction. Again, the difference in effectiveness in that case is not statistically significant. It is important to note here that while the reported reduction in the response time is in terms of milliseconds in this in-lab experiment, such speedup is highly and typically needed during Web search tasks (such as searching for background sources over the Web) in real-time online scenarios.

How far can we reach with reranking? (RQ3)

We observe that the most effective method reported for background linking, including the baseline method, is still far from being optimal (maximum reached effectiveness is nDCG@5 = 0.53). But what if a method retrieves relevant background links at lower ranks than top 5? Those links will not be considered when computing the nDCG@5 score (i.e., will not be shown to the user). However, this initial set of retrieved links can be re-ranked by another downstream method (e.g., by fine-tuning pre-trained models) to further improve the effectiveness. To measure how far any of the keyword extraction methods can potentially reach in terms of nDCG@5 with post hoc reranking, we retrieve, using each method, a set of N candidate articles, then we optimally rerank those articles using the judgement scores assigned by TREC annotators. Not-annotated articles are considered non-relevant, and thus are assigned a 0 score. Doing so, we obtain oracle runs for the different methods. We then compute nDCG@5 for those runs.

Figure 9 illustrates the performance of the oracle runs, changing N from 10 to 100, for all keyword extraction methods, besides the baseline method. We first notice that the performance can reach 0.84 at N = 100, indicating the great potential for reranking the top 100 retrieved articles. Moreover, we can notice that the k-Truss and k-Core methods exhibit the best performance (i.e. obtaining the most promising lists), followed by TF and TF-IDF. That suggests another criterion of selecting a keyword extraction technique for our task, which is the potential for better effectiveness with postdoc reranking. We also note that the baseline method has the lowest reranking potential, indicating that keyword extraction is not only needed for efficiency purposes, but also needed for increasing the potential of having more relevant background articles at the top of the retrieved list.