Recent achievements in high-throughput sequencing technologies (Goodwin, McPherson & McCombie, 2016; van Dijk et al., 2018) have led to the generation of massive amounts of genomic data (Gao et al., 2019; The ICGC/TCGA Pan-Cancer Analysis of Whole Genomes Consortium, 2020) available for the research community. Many omics databases have been developed and have collected freely accessible datasets (Molkenov et al., 2019; Rigden & Fernández, 2020) for analysis by the bioinformatics community. Modern bioinformatics tools and methods are in high demand for analyzing and interpreting the big omics data generated by the different types of multi-omics platforms available. Different standard data types and file formats have been developed to store and analyze sequence and genomics data. Variant Call Format (VCF) (Danecek et al., 2011) is a tab-delimited text file format that is often used in bioinformatics to store genomic variants. A VCF file consists of the header, including meta-information lines and field definition lines (column names), and the body (data section). An arbitrary number of meta-information lines start with ’##’ and provide a description of the VCF file. The body of the file consists of eight mandatory columns: chromosome (CHROM), starting position of a variant (POS), variant identifiers (ID), the reference allele (REF), a list of alternate alleles (ALT), a PHRED-scaled quality score (QUAL), filter information regarding variant validity (FILTER), and annotation information (INFO). Additional columns describing samples can also be added. Each row of the file describes specific genomic variants (SNVs, INDELs, CNVs, and other structural variants) at the given chromosome and genomic position.
VCF files often store information about numerous samples and can therefore reach huge sizes–gigabytes, or sometimes terabytes. This creates an issue for the readability of VCF files and further analysis for non-programmers, as manual data extraction and analysis using Microsoft Excel or other table processing software may not be possible due to the RAM capacity limitation of standard computers. We introduce re-Searcher, bioinformatics tool specifically developed for simplified mining and analysis of big-size VCF files. We developed a multi-platform user-friendly graphical user interface (GUI) tool for offline access, while re-Searcher web application can be used online via web browser. re-Searcher has been developed for the broader biomedical community and solves the problem of working and analyzing genomics data stored in VCF format.
Materials & Methods
The re-Searcher application was written in Python 3 (Van Rossum & Drake, 2010) with the implementation of the Tkinter (Python Software Foundation, 2020) package to build the GUI, and the Pandas (McKinney, 2010) library to extract columns. For convenience of users who would like to build re-Searcher into their pipelines command line interface (CLI) is available as python script (see Availability). CLI mirrors functionality of GUI regarding files processing. We developed web version of re-Searcher for users to manipulate VCF files without downloading CLI or GUI versions of re-Searcher. The web application was developed using Django web framework (Django Software Foundation, 2013), to run python script via WSGI on Apache web server (Fielding & Kaiser, 1997). The web version runs re-Searcher scripts on server and takes input files from website and returns processed files to user to download.
re-Searcher solves the problem of analyzing large VCF files by not loading the whole file directly into RAM, but instead pre-processing it in chunks and utilizing a simple and intuitive interface (Fig. 1). The main advantage of re-Searcher in comparison with other tools is the presence of a simple and user-friendly interface, GUI and web interface, instead of a CLI, as well as a lack of confused installation procedures typical for existing tools. The generalized workflow of re-Searcher consists of several steps: selecting an input file, setting up necessary filtering parameters, data processing, and exporting a filtered output VCF file (Fig. 2). re-Searcher browses and opens VCF files with extensions “.txt” or “.vcf”, before performing the following filtering and extraction options:
VCF files can be large and, if a user needs to know only certain information in a file header (e.g., a particular meta-line or sample ID), the software can extract only the header from the original VCF file and save it into a new file.
If genomic variants need to be filtered according to the presence of a keyword, the software can find these rows and extract them into a new VCF file. Users may input multiple keywords by accessing the entry field or by uploading a .txt file with keywords.
If the user needs only particular sample IDs in a VCF file, the software can extract the necessary sample columns into a new file. Similar to a keyword search, users may input multiple sample IDs by accessing the entry field or by uploading a .txt file with the IDs.
Genotype format conversion
re-Searcher can convert numeric genotype (GT) format into letter format. The conversion option is one of the most used operations when working with VCF files, for example, in further comparative analysis of genetic variants or SNPs. The original GT format in VCF files is numeric (0/0, 0/1, 1/1 for biallelic sites or 1/2, 2/3, etc. for multiallelic sites), where 0 is a reference (REF) allele, 1 is a first alternative (ALT) allele, 2 is a second ALT allele and so on (Danecek et al., 2011; (Campbell et al., 2016). After GT format conversion REF number is replaced with REF letter and ALT number with corresponding ALT letter (Fig. 3). For instance, if GT of first sample is 0/1 and GT of second sample is 1/2 in numeric format, while REF and two ALT are GC, T and CAA respectively, then after conversion first sample’s letter GT becomes GC, T and second sample’s letter GT becomes T, CAA.
The final output file is a filtered and processed VCF file and is generated with a complement log file containing the file processing information, name of specified file and work directory with outputs.
Results and Discussion
We have compared the main features of re-Searcher with other existing and open source tools VIVA (Tollefson et al., 2019), VCFtools (Danecek et al., 2011), GEMINI (Paila et al., 2013), BrowseVCF (Salatino & Ramraj, 2016), VCF.Filter (Müller et al., 2017), VCF-Miner (Hart et al., 2016) and prepared a detailed table (Table 1). The compared tools have been developed and implemented in different programming languages (Julia, C++, Perl, Java and Python) and dedicated libraries. VCFtools is one of the most cited and advanced tools for processing VCF files, but it requires additional computational skills for effective usage. VIVA is the only tool that provides the possibility for advanced visualization and plotting figures. In addition to re-Searcher, other tools with a GUI available for users are BrowseVCF, VCF.Filter, and VCF-Miner, while web interface is available in only in re-Searcher (Fig. 4) and GEMINI. From these, only re-Searcher, BrowseVCF and VCF.Filter tools support multiple operational systems (Windows, MacOS, Linux). Searching the whole VCF file by keyword and the corresponding extraction of data based on keywords are features available on re-Searcher and BrowseVCF, whereas genotype conversion is a unique feature of re-Searcher.
|Technical Aspects||Compatibility with operation system||Windows, MacOS, Linux||Windows, MacOS, Linux||Windows, MacOS, Linux||Windows, MacOS, Linux||Windows, MacOS, Linux||Windows, MacOS, Linux||Windows|
|Interface||GUI, Web Browser, CLI||CLI, Jupyter Notebook||CLI||Web Browser, CLI||GUI, CLI||GUI||GUI|
|Functionality||Search by keyword||V||X||X||X||V||X||X|
|Genotype format conversion||V||X||X||X||X||X||X|
|Export filtered VCF file||V||X||V||X||X||V||V|
re-Searcher is a multi-platform tool and can be run on MacOS, Windows and Linux operating systems. Performance of re-Searcher has been evaluated on these PC platforms (Windows 10 Pro OS and Linux Mint 17 OS-based PC: CPU Intel Core i5-8250U 1.80 GHz, RAM 4Gb and MacOS Catalina based PC: CPU Intel Core i7, 3.2 GHz, RAM 8 Gb) with different VCF file sizes. Different size VCF files (0.081 Gb, 0.814 Gb, 1.320 Gb, 1.980 Gb and 7.950 Gb) were used as input datasets for evaluating re-Searcher performance. The results of the performance benchmarking are shown in Table 2.
|OS||VCF file size (Gb)||Execution Time (sec)|
|Extract header||Keyword extraction||Sample ID extraction||GT
We have used big VCF files from the 1000 Genomes Project (The 1000 Genomes Project Consortium, 2015) and then generated the different sized testing VCF files from this dataset. The Linux-based systems had the fastest execution time for different operations in comparison with Windows and MacOS systems.
re-Searcher executable software including source code, tested VCF files and additional information are publicly available on the GitHub repository https://github.com/LabBandSB/re-Searcher. re-Searcher is free bioinformatics tool and open to all users without login and registration requirements and do not require an installation of additional tools. CLI version of re-Searcher is also available on the GitHub repository for incorporation into other pipelines. In addition, web version of re-Searcher is available at https://nla-lbsb.nu.edu.kz.
Exploring and analyzing VCF files generated after the bioinformatics processing of sequencing data is one of the important steps performed by researchers during analysis and meta-analysis of genotype/phenotype associations. We have developed and introduced an easy-to-use bioinformatics tool, re-Searcher, with several unique features for mining big VCF files and realized with a simple graphical user interface and web interface that makes it easily available for clinicians and researchers without any computational skills. Several improvements such as visualization options (clustering and plotting functions) with Principal Component Analysis and heatmap methodologies are under future development of re-Searcher.