Biblio-MetReS for user-friendly mining of genes and biological processes in scientific documents

Organism selection

Biblio-MetReS is organism-centric. Users must select their organism of interest from a list of more than 1200 organisms with fully sequenced and annotated genomes before starting any search. They must also decide whether they want to perform GO and/or pathway term co-occurrence analysis. After these decisions are made, the program loads the genes for the organism from the program’s central database. If selected, terms from the GO classification, KEGG and/or Panther pathways are also loaded into the application’s front end.

Document analysis

To analyze networks of co-occurrence Biblio-MetReS needs users to select at least one gene from their organism of interest and at least one database in which to perform document analysis. The seed list of genes is then used to identify relevant documents in the selected database(s) of documents. The text in the flagged documents is then analyzed to identify additional genes from the organism of interest, as well as GO and/or Pathway terms. This allows users to identify co-occurrence among GO/Pathway entities and between GO/Pathway entities and gene/protein entities in sentences, paragraphs or documents. For further description of this procedure, please see section 1 of the Supplementary Materials.

Biblio-MetReS uses exact matching of gene names to an internal dictionary of synonyms to identify co-mentions of genes/proteins in the text of scientific records. The gene synonyms are those officially defined by NCBI. For any given gene, all synonyms are searched for in the text of flagged documents. Similarly, Biblio-MetReS uses exact matching of GO terms to an internal dictionary of synonyms defined by the gene ontology consortium (Gene Ontology, 2013) to identify co-mentions of GO terms in the text of scientific records. The same exact matching is done to identify mentions to entities from the complete joint sets of KEGG (Kotera et al., 2012) and Panther pathways (Mi et al., 2005).

Co-occurrence of any two terms is analyzed at three levels. First, all possible pairs of terms are searched for in each document as a whole. Then, each document is divided into paragraphs and the pairs of terms identified in the document are searched for within each paragraph. Finally, each paragraph is divided into single sentences and the pairs of terms identified in that paragraph are searched for within each sentence. Within each level, the distance between each term in the pair is not taken into account.

The database containing the organisms and their gene names, as well as the GO terms, is updated every three months using information compiled automatically from NCBI and GO.

Calculating the significance of term co-occurrences

To attribute statistical significance to the co-occurrence of a pair of genes, pathway terms, GO terms, gene-pathway terms, gene-GO terms or GO-pathway terms, we calculate several metrics. First, we measure how frequently the different pairs co-occur in sentences, paragraphs and/or documents. We then take the odds ratio of the frequency of occurrences in the first two categories with respect to that of the third. The closer to one these odds ratios are, the more frequent it is that both genes are mentioned only in the same sentences or paragraphs of a document, rather than appearing haphazardly in different sections of the text.

Second, we calculate how much information we gain by having two terms, T_i and T_j, co-occur, when compared to the individual occurrences of the terms. To estimate this we use information theory. The individual probability of occurrence of a term is denoted as p(T_i) and it is formally defined as p(T_i) = a∕n, where a is the number of documents where Term T_i appears, and n is the total number of documents. The joint probability of co-occurrence of two terms, p(T_i, T_j), is defined as p(T_i, T_j) = b∕n, where b is the number of documents where terms T_i and T_j simultaneously appear, and n is the total number of documents. The mutual information, MI(T_i, T_j), is then calculated as follows: $\begin{array}{ccc}\mathrm{MI}\left(T_i,T_j\right)=p\left(T_i,T_j\right)\times log\left(p\left(T_i,T_j\right)∕\left(p\left(T_i\right)p\left(T_j\right)\right)\right)& & \end{array}$

Finally, and in order to attribute some form of statistical significance to the co-occurrence of a pair of terms, we do as follows. Consider a set of n sentences (paragraphs, documents) [1, …, n]. For a given term k define $\begin{array}{ccc}{y}_{i,k}=\left\{\begin{array}{ccc}1\hfill & \Leftarrow =\hfill & \text{term}k\text{occurs}\in \text{sentence (paragraph, document)}i\hfill \\ 0\hfill & \Leftarrow =\hfill & \text{otherwise}\hfill \end{array}\right.& & \end{array}$ Now, for terms k₁ and k₂ define φ_{k1, k2} = y_{i, k1×}y_{i, k2} which has value 1 when both terms co-occur and 0 otherwise.

Both these variables have a Bernoulli distribution. If the occurrence of terms k₁ and k₂ is independent, then p(φ_{k1, k2}) = p(y_k1)p(y_k2) would be expected, where p(y_{k, ⋅}) is the relative frequency of occurrence of term y_{k, ⋅} and p(φ_{k1, k2}) is the relative frequency of co-occurrence of terms k₁ and k₂ in the total number n of sentences (paragraphs, documents). Then, a Pearson statistic can be used to test for independence of occurrence between k₁ and k₂ by comparing the observed frequencies, n₁ = n × p(φ_{k1, k2}) and n₂ = (1 − p(φ_{k1, k2})) × n, with the expected frequencies under the null hypothesis of independence, which would be m₁ = n × p(y_k1) × p(y_k2) and m₂ = n × (1 − p(y_k1) × p(y_k2)). The Pearson statistic is computed as follows $\begin{array}{ccc}{X}^2=\sum _{i=1}^2\frac{{\left(n_i-m_i\right)}^2}{m_i}& & \end{array}$ This statistics follows a chi-square distribution with one degree of freedom, i.e. ${\chi }_1^2\sim X^2$; hence, the p-value can be calculated as $p=Pr{\chi }_1^2>{\chi }^2$ to assess whether the observed co-occurrence is higher than the one expected by pure chance.

Precompilation strategy

Biblio-MetReS v2.0 implements a precompilation approach that works in the following way. Any search done will identify a given number of documents in the database(s) selected by the user(s), for an organism of interest. If a given document has not been found in any previous search by any user, in the context of that organism, Biblio-MetReS will analyze it as described in section 1 of Supplementary Materials and all information contained in that document and relevant for the analysis will be stored in a central database (see section 3 of Supplementary Materials for detailed information). If a given document has been previously found by any user, its information will be directly accessed from our central database, and the document will not be reanalyzed. This means that newly found documents are mined on the fly by the program to find and count mentions of relevant entities, while mentions in documents that have been previously found are simply looked up in our central database.

Biblio-MetReS and Biblio-MetReS player

Biblio-MetReS v2.0 can be used to identify genes/proteins from more than 1200 different organisms in records stored in a variety of databases. Users download the application and run it locally. A functioning internet connection and a local copy of JAVA are required for the program to work. Upon starting the program, users log into the central Biblio-MetReS database and choose which organism they are interested in and whether they want to search only for co-occurrence of genes/proteins or if they also want to include biological pathways and/or GO biological processes in the analysis. Once this choice is made, the program loads the necessary information from the central database. Taking this approach, instead of including all the data locally in the program installation, permits making an application that is much smaller in size and needs less RAM to function properly. Subsequently, users select the source of documents that they want to analyze, as well as the genes/proteins and/or pathways/GO biological processes that they want to search for. They can also include their own handpicked list of processes to be searched. Once the search is launched, Biblio-MetReS will identify documents in the relevant databases that contain mentions to the relevant search items. After identifying these documents, the application fully analyzes them to identify mentions for any additional gene or Pathway/GO biological process via a dictionary matching approach. The co-occurrence of the different entities is analyzed at the level of the whole document, of individual paragraphs and of individual sentences, and the significance of this co-occurrence is calculated as described in Usié et al. (2011). The information that is relevant for the co-occurrence calculations is stored in the central Biblio-MetReS database. Any subsequent searches that identify the same document will not reanalyze it; instead, these numbers are directly retrieved from that database. Once the analysis is complete, the users can visualize it in graphical and textual form. Links to the documents and sentences where co-occurrences are found are provided. Graphical visualization of the results can be done in different ways. Users can create graphs for the global co-occurrences network and for gene- or pathway/process-centric co-occurrences at the document/paragraph and sentence levels. Significance and Mutual Information of each co-occurrence is also provided in tabular form. The graphical representation of the networks is automatically stored in local xml files. These files can be opened using a small app, Biblio-MetReS Player, which can be downloaded from the Biblio-MetReS website. This permits reviewing previously obtained networks without having to redo the search. All this process is summarized in Fig. 1.

Comparative performance and benchmarking of new types of entities

The benchmarking of the program and its improvements with respect to version 1.0 was carried out using four organisms of interest: Saccharomyces cerevisiae, Homo sapiens, Escherichia coli and Drosophila melanogaster. For each organism we used as a search seed a set of genes belonging to Glycolysis, Lysine metabolism and RNA processing pathways. The genes were chosen to reproduce the experiments reported in Usié et al. (2011). Details are shown in Table S1. We benchmarked three different aspects of Biblio-MetReS. First, we benchmarked the comparative identification of genes between Biblio-MetReS, iHOP, and STRING, given that these three applications have similar target audiences. Second, we benchmarked the ability of Biblio-MetReS to identify biological processes/pathways. Finally, we benchmarked the improvements in Biblio-MetReS run time made by implementing the combined pre-processing/live analysis strategy.

Benchmarking the comparative identification of genes between Biblio-MetReS, iHOP, and STRING was done in the following way. We used the genes and organisms described in Table S1 to interrogate independently Biblio-MetReS, iHOP, and STRING. For each of the three applications, the complete set of results for each gene from the same pathway were pooled together for analysis. The results from STRING were further filtered to eliminate all genes and interactions that were not literature based. Table S2 compares the results for the three applications. In summary, Biblio-MetReS find the largest number of genes, followed by STRING, and iHOP. The number of genes found by STRING and Biblio-MetReS are of the same order of magnitude, while iHOP finds between one and two orders of magnitude less genes. This result derives from the fact that iHOP analyzes only Medline abstracts, while Biblio-MetReS and STRING analyze the full text of Pubmed publications, in addition to the Medline abstracts. This is confirmed by the fact that, when Biblio-MetReS is run only to analyze Medline abstracts, it finds a similar number of genes as iHOP (data not shown). As was observed in Table S3, the genes found by each of the three applications for the same experiments only partially overlap and this is explained by the different datasets analyzed by each of the programs and by partially different dictionaries of gene synonyms (Usié et al., 2011).

Neither STRING nor iHOP permit identifying GO terms and their associations to genes. Therefore we cannot perform experiments that are similar to the comparative benchmarking experiments described above. In light of this, benchmarking of Biblio-MetReS’ ability to identify biological processes/pathways was done in the following way. To perform the GO identification benchmark we used the test and development sets of the BioCreAtIvE IV GO task corpus [3]. We used Biblio-MetReS dictionary matching approach to identify GO terms in the non-annotated documents and then analyzed the corresponding annotated documents. We found that Biblio-MetReS identified 100% of the annotated GO terms in both sets (2963 terms in the training set and 2243 terms in the development set). Biblio-MetReS also identifies 2259 additional GO terms in the development set and 2119 additional GO terms in the training set. For the purpose of our testing these terms must be considered false positive. Taking this into account, the precision in the development set is 50%, while in the training set it increases to 58%. The F-score performance of Biblio-MetReS is 33% in the development set and 37% in the training set, which is on par with the best approaches presented in the lattest BioCreAtIvE IV challenge (Mao et al., 2013).

Benchmarking run time was done in two ways. First, we search only the Pubmed database. Second, we search by selecting all literature databases available in Biblio-MetReS. In both tests we used all the seed genes from Table S1. Each seed is used by Biblio-MetReS as a query search. This query search is launched twice. When the first search is done there are no preprocessed documents in Biblio-MetReS’ database. The information in documents is analyzed on-the-fly and stored. Then the searches are repeated, now with the documents stored in Biblio-MetReS’ database. This allows us to estimate the percentage of run-time saved by preprocessing the documents. The results are shown in Fig. 2 and in Table S2 of the supplementary materials. On average we get decreases in run time of more than 90%.