Hand-based multibiometric systems: state-of-the-art and future challenges

Rationale for the survey

There is a wide range of biometric traits that can be used for authentication purposes including face, hands, iris, retina, etc. All these traits offer their own advantages and drawbacks and most of them are thoroughly covered in the literature. In this paper, we aim to focus on the existing hand based multibiometric systems explored over the past five years. The use of hand modalities offers several advantages over others: they are highly accurate for recognition, generally make use of inexpensive technology, are fast for matching and require templates of very small sizes, resulting in a small memory footprint and are less sensitive to imaging conditions. Moreover, the hand-based modalities are more robust since they are not affected by emotions and other behavioral characteristics of the individuals such as tiredness, stress, etc. Given this, it is clear that with respect to some specific aspects, the use of hand based modalities is superior as compared to others for biometric authentication.

There is a large volume of literature discussing about the use of hand based modalities for authentication but the work becomes limited as it is directed towards multimodal systems. This domain was constrained until recently due to the issues related to the power consumption, size, and cost, etc., of the hardware required for executing the biometric systems. However, over the recent years, the revolution in the hardware design industry has led to miniaturized devices having tremendous capabilities, enabling the developers to execute multiple systems on very light, low-powered, and small devices with significantly lesser cost. Consequently, it has been possible to implement multibiometric systems very efficiently, triggering a lot of research in this direction. Moreover, to the best of our knowledge, a thorough survey of hand based multibiometric systems, their effective usage in biometric authentication and the main challenges faced, is currently not available in the literature.

With regard to the security of the biometric system, being multibiometric in nature adds itself another layer of security even though there are multiple points of attack on an authentication system. We aim to tackle the literature available regarding security of multibiometric templates. The rationale for focusing on security of multibiometric template is that they lead to a 3-dimensional vulnerability to a biometric system in contrast to their counterparts (Jain, Nandakumar & Nagar, 2008): 1. Template can be replaced by an imposter to gain unauthorized access, 2. A spoof can be created from the template to aide in unauthorized access, and 3. The stolen template can be replayed to the matcher to gain access. Therefore, it is vital to protect the templates from an adversary; unlike PINs and passwords, a biometric template if compromised cannot be revoked and reissued, so considering the criticality in this context, we aim to deal with the research contributions that are devised to protect the integrity of the saved templates.

This paper presents a systematic review on the use of hand based multibiometric systems and an analysis of their efficacy in performing authentication. In this context, the main contributions are as follows:

• Discussing the main advantages and motivations behind the usage of hand based multibiometric systems.

• Presentation of a taxonomy to categorize the literature with respect to the two parameters: authentication and template protection.

• Presentation of a summary of literature on the above-mentioned parameters, with critical/brief comments.

• A discussion about open issues and the direction of future work on hand based multibiometric systems.

In summary, this paper covers two major directions of work on hand based multibiometric systems: 1. The work on various schemes to perform feature extraction and authentication of individuals using multibiometric systems based on hand modalities. The main objective of such studies is to ensure that the performance metrics of the systems are very good and they can be effectively used, and 2. Once the templates have been acquired from the users (the users are enrolled), how these templates can be effectively archived such that they are not susceptible to attacks.

Paper organization

The rest of the paper is organized as follows: In “Survey Methodology”, we present the methodology followed in this survey. We discuss an overview of a biometric system, multibiometric system and its types in “Overview of a Biometric System”, followed by the discussion on fusion methods used in a multibiometric system in “Fusion Methods”. Later, we talk about different hand-based modalities that are used for biometric authentication in “Hand Based Modalities”. Then, we discuss the feature extraction methods in existing hand multibiometric systems in “Feature Extraction Techniques for Hand Multibiometric Systems” and methods used to perform multibiometric template security in “Multibiometric Template Security”. Finally, we discuss about various open issues and challenges in the topics covered in “Open Challenges and Future Directions” and conclude the paper in “Conclusions”.

Unibiometric systems

Traditionally, biometric recognition systems are unibiometric, which employ a single biometric trait for authentication purposes. It may use one of the physical or behavioral biometric traits, such as fingerprint recognition (Maltoni et al., 2009; Hong, Wan & Jain, 1998; Yuan, Sun & Lv, 2016), face recognition (Zhao et al., 2003; Masi et al., 2018; Wang et al., 2018), iris recognition (Nguyen et al., 2017), signature, etc. In the literature, the use of unibiometric systems is widely employed with very good recognition results. However, such systems are typically constrained due to several factors including lack of accuracy due to noisy data, non-universality of biometric traits for registration, physiological limitations of biometrics, and vulnerabilities in biometric systems (Table 1) (Dinca & Hancke, 2017; Oloyede & Hancke, 2016).

Some biometric modalities are more vulnerable to some specific problems, e.g., spoofing a fingerprint is relatively easier as compared to a vein/palm pattern. However, the recognition accuracy of fingerprints is far more superior. These are complementary properties of two different biometric modalities, which can be exploited together in a multimodal biometric system, hence making the system more tolerant to spoofing while maintaining a higher accuracy.

Multibiometric systems

When using the unibiometric systems, we may encounter problems due to several issues including, but not limited to, missing data (Nandakumar, Jain & Ross, 2009) (e.g., occlusion in face image), poor sampling (Grother & Tabassi, 2007), biometric duplication (Sudhish, Jain & Cao, 2016), low discrimination among samples (e.g., hand shape/geometry) between distinct users, vulnerability to attacks, and spoofing, etc (Jain & Kant, 2015). In situations like these, it may be necessary to make use of multiple biometric cues to boost the accuracy of a recognition system. The multibiometric systems offer so many features, making them more convenient and feasible as compared to the unimodal systems. There can be different sources of biometric information in a multibiometric system due to which such systems can be classified into the following five major categories (Fig. 3):

Performance metrics for evaluation

Multiple metrics can be employed to assess the performance of a biometric authentication system. Choosing a particular metric(s) depends upon the nature of evaluation. Following are the basic raw metrics and their descriptions:

• True Accept (TA): A genuine user is correctly verified to its corresponding template stored within the biometric system.

• True Reject (TR): An imposter is correctly rejected as its data does not match any template stored within the biometric system.

• False Accept (FA): An imposter is incorrectly verified as a genuine user as his data is matched to the template stored within the biometric system.

• False Reject (FR): A genuine user is incorrectly rejected as his data does not match any template stored within the biometric system.

The standard metrics that have been used to evaluate the performance of the authentication system in the literature are as follows:

• False Accept Rate (FAR): Describes the percentage of impostors that were incorrectly verified as genuine users. It is calculated on the basis of the following formula:

  (1) $$FAR={\displaystyle \frac{FA}{FA+TR}}$$

• False Reject Rate (FRR): Describes the percentage of genuine users that were mistakenly rejected from a biometric system. It is calculated on the basis of the following formula:

  (2) $$FRR={\displaystyle \frac{FR}{TA+FR}}$$

• Correct Recognition Rate (CRR): It gives the probability that the system will correctly identify the input template from the templates in the database. It is given by the formula:

  (3) $$CRR={\displaystyle \frac{TA}{TA+TR}}$$

• Genuine Acceptance Rate (GAR): Describes the percentage of genuine users accepted by the biometric system. It is given by the formula:

  (4) $$GAR=100-FRR$$

• Accuracy: It is the ratio between verified cases (both True Accept and False Accept) to all possible cases. It is given by the formula:

  (5) $$Accuracy={\displaystyle \frac{TA+FA}{TA+FA+TR+FR}}$$

• Equal Error Rate (EER): Describes the point at which FAR and FRR are equal. Smaller values of EER refers to improved performance of a biometric system.

Sensor level fusion

In sensor level fusion, the raw information is gathered from various sensors and is fused at the initial level prior to feature extraction to produce a raw fused information. Fusion of two images can take place at pixel, feature or at signal level. Fusion at sensor level can be between multiple samples of the same biometric gathered from multiple sensors (Yang et al., 2005) or multiple instances of the same biometric taken from a single sensor (Othman & Ross, 2012). Relatively less research has been done on this type of fusion in biometrics.

Feature level fusion

In feature level fusion, the features extracted from multiple biometric sources are combined together in the form of a single feature vector. In this fusion technique, features from different sensors, samples, and traits can be combined together. At this level of fusion, signals from various biometric channels are firstly pre-processed and their feature vectors are calculated independently; by using fusion algorithms, the feature vectors are fused to form a combined feature vector, which is used for recognition (Xin et al., 2018). The incorporation of multimodal biometric traits in this type of fusion can be employed to exploit specific strengths of different biometric modalities (Nagar, Nandakumar & Jain, 2011; Jagadiswary & Saraswady, 2016; Prasanalakshmi, Kannammal & Sridevi, 2011). Although better recognition results can be expected using this type of fusion technique, it has certain limitations including the lack of compatibility of different biometric features, curse of dimensionality, etc.

Matching score level fusion

This type of fusion is done by joining the scores yielded by the matching module of each feature vector with the template. The features are processed independently along with the calculation of scores, followed by the calculation of composite matching scores (He et al., 2010; Yilmaz & Yankoglu, 2016; Kabir, Ahmad & Swamy, 2018). This is done by the checking the confidence scores, which are obtained using each feature vector. This type of fusion technique is typically easy and thus is being used by different multibiometric systems for effective execution.

Rank level fusion

In this type of fusion, sensor data is acquired followed by the feature extraction. The matching of this feature vector is performed against all the available templates in the database and similarity scores are obtained (Kumar & Shekhar, 2010; Monwar & Gavrilova, 2009). The scores are arranged in the descending order and the entry corresponding to the lowest rank (indicating similarity of feature vector with the respective template) is taken as the most relevant to the data that is acquired from the sensor. The rank level fusion can also be employed for multibiometric traits and thus can yield a recognition score with a higher confidence. However, it should be noted that in addition to the pre-processing of sensor data, additional computational load is transfered on the matching module. Therefore, the rank level fusion can be computationally very complex especially when more than one biometric trait is employed.

Decision level fusion

In this type of fusion, the information obtained from different decision modules is combined together to decide about the identity of a user (Jiang et al., 2014; Niu et al., 2008). The recognition results of each biometric trait are individually obtained followed by a fusion of these decisions to obtain a final decision regarding recognition (Li et al., 2018; Ghosh, Sharma & Joshi, 2014). Various methods to perform this type of fusion can be used, e.g., majority voting can be employed (Jimenez, Morales-Morell & Creus, 1999). In systems which require enhanced security and fail safe functioning, rule based decision can also be made such as the use of a logical ‘AND’ operation, indicating that it is necessary for all biometric traits to be yielding the same output.

Fingerprint

Fingerprint is one of the most established biometric modalities due to its high recognition rates and consistency, and has been in existence for over a century. The ease to acquire fingerprints and their wide usage has led to many commercial applications relying on them as far as biometrics are concerned. A fingerprint is formed by the coexistence of a collection of ridges and valleys, thus yielding a pattern, which is distinct for different human beings. These patterns are also referred to as “minutiae” and are mainly composed of bifurcations, enclosures, ridge endings and ridge dots. Further, the minutiae are subdivided into sub minutiae such as pores, crossovers, and deltas (Fig. 6). A fingerprint biometric system has four main stages: acquisition of data, feature extraction, template creation, and matching. The ease of use and a small space required for the storage of template has made it one of the best biometric technologies to employ commercially.

On fingerprint biometric, both quantitative and qualitative works exist. A survey of around 160 users was done in Arteaga-Falconi, Al Osman & El Saddik (2015) and Cappelli et al. (2007) in which the users gave a positive response towards using this technology for smartphones. Furthermore, various technological contributions presenting quantitative results show that a fingerprint take less than one second for matching, achieve 0.07% EER on a database of 100 subjects, false rejection rates of upto 0.04% and false acceptance rates of upto 0%, 4.18% and 8.91% using three confidence coefficients, i.e., 99.0%, 99.5% and 99.9% respectively. These results indicate a very high recognition performance in a very small amount of time, promoting the use of fingerprint technology for real time implementation of systems requiring biometric validation.

Palm print

Palm print is a popular biometric modality, which has attracted the attention of many researchers. However, it is a relatively new biometric modality as compared to its counterparts, such as face, fingerprints, etc. A palm print image consists of some rich intrinsic features such as ridges and palm lines, delta lines, principle lines, minutiae features, wrinkles, etc. (Chen, Huang & Zhou, 2013; Huang, Jia & Zhang, 2008) that are deemed to be permanent and unique for every individual (Fig. 7). Owing to these inherent features, palm prints generate unique biometric characteristics for every individual that are reliable for identification purposes (Zhang, Zuo & Yue, 2012; Zhong, Du & Zhong, 2019). The main issue that is responsible for reducing the performance of palm print systems is the deformation of images during the image acquisition process. Attempts are being made to solve this issue by using contact devices, but researchers have faced several challenges in the design of such devices including its size and limited usability, along with several challenges including position, stretch and rotation of the palm print. Lately, researchers are resorting to contactless devices again with low resolution imagery used for commercial application and high resolution imagery for applications such as criminal investigation.

Some of the most recent contributions on palm print biometric (Zhang et al., 2017; Tabejamaat & Mousavi, 2017) show that the recognition rates of upto 98.78% and 97.2% respectively have been achieved within processing times in the order of milliseconds on databases of fairly large size (about 12,000 instances). The most relevant contribution in this regard with the best accuracies are presented by Luo et al. (2016) in which the authors have reported an accuracy of 100% on a dataset having 4,600 instances of palm prints. This is in contrast to a general perception that palm prints are not as accurate as fingerprints. However, it should be noted that the possibility to obtain a relatively large dataset to validate the findings on fingerprints having a statistically higher significance is much more likely in contrast to palm prints for which the availability of dataset is relatively limited.

Veins

Vein biometrics, also known as vascular biometrics, refer to a biometric system that measures parts of an individual’s circulatory system for identification. Vein pattern recognition technology has gained a significant attention due its unique attributes along with liveness property yielding very high recognition rates. Vein patterns are segmented into different sub-modalities amongst them most commonly used come from the palm (Zhou & Kumar, 2011), palm dorsal (Joardar, Chatterjee & Rakshit, 2014), wrist (Pascual et al., 2010), or finger (Lee et al., 2010). The sub-dermal nature of veins makes these types of biometrics a highly secure modality (Crisan, 2017). In a vein biometric system, image acquisition is carried out by using near-infrared (NIR) imaging device. The NIR light maps the vein locations, because the hemoglobin in veins absorbs the NIR light. A high contrast image is created by visualization of the vein pattern as shadows appearing over a white background. These high contrast images with vein patterns are used for recognition using various texture feature extraction techniques.

A quick review of the literature elucidates the facts that very high recognition rates are obtained on this biometric trait. Researchers have reported an accuracy of upto 99.4% reinforcing the theoretical claims of high uniqueness of vein patterns (Das et al., 2018). However, the requirement of using sophisticated acquisition devices for obtaining the biometric data makes this modality relatively less popular (Kilian et al., 2020). Moreover, using vein patterns can be a challenge in some cases because of the physiological changes taking place due to ageing and various medical conditions (Oloyede & Hancke, 2016).

Hand geometry

Hand geometry/shape is a very simple biometric technology that uses the measurements of human hand to verify the identity of the individuals. The measurements include the length, shape and width of fingers and size of palm (Fig. 8). The biometric systems employing hand geometry are widely used as they have a high public acceptance (Babich, 2012; Jain, Dass & Nandakumar, 2004, Sharma et al., 2015). However, it should be noted that the systems based on this technology are not scalable as the hand geometry is not highly unique (Oloyede & Hancke, 2016). Nevertheless, it is widely used at places providing access control, where the main objective is to find out if someone is illegally trying to gain access to someone’s personal identification. A hand reader guarantees that a worker is actually available at a place where he is meant to be. It is also used for implementing time attendance of the employees and helps in stopping the employees from buddy punching (which takes place commonly with fingerprint technology). Hence, the payroll accuracy of a company is guaranteed with a higher probability when hand geometry is used (Babich, 2012).

Due to the lack of ability to differentiate between the people effectively, the usage of hand geometry is somewhat limited and typically used in conjunction with other biometric modalities for improved recognition rates. Some recent contributions on hand geometry show that an EER of upto 0.31% has been achieved by Sharma et al. (2015) with upto 50 distinct users. A novel contactless sensing system (Kanhangad, Kumar & Zhang, 2011) based on multi sampling has been proposed, which has been used to authenticate a database of 100 people representing upto 200 hands with about 50% improvement in the recognition rates (Oloyede & Hancke, 2016). Nevertheless, the technology is not as accurate as its counterparts and thus is not very useful in a standalone setting for large scale deployment for commercial purposes.

Finger knuckle print/inner knuckle print (FKP)

Finger knuckle print is one of the emerging hand based modalities used for biometric verification of the individuals (Kumar & Ravikanth, 2009). The finger knuckle patterns can be easily acquired using contactless devices. In contrast to the more established modalities such as fingerprints requiring high resolution imagery, the knuckle patterns can be easily captured using low-resolution samples (Zhang, Lu & Zhang, 2018). Additionally, the patterns on the outer surface of the knuckle appear at an early stage and survive for a longer periods of time and are specifically useful for the workers, labourers, cultivators, etc., whose fingerprints are more susceptible to damage due to the nature of work (Yang, Yu & Liao, 2009). In a biometric system based on finger knuckle, the physiology which differentiates two different people is due to the lines, creases and texture of the knuckle print that lie at the three knuckle joints of the fingers (Jaswal, Kaul & Nath, 2016) (Fig. 9). These lines appear before birth and rarely change over an individual’s lifetime.

The knuckle print is a biometric that can be acquired without any physical contact with any sensor. Therefore, the chances of spoofing are significantly reduced. These are highly stable for individuals from various age groups; however, their widespread usage is still not reported. A quick survey of the literature shows that researchers have obtained a high accuracy on the identification of persons using knuckle prints with an overall accuracy of upto 98% in real time on a dataset of size 7,900; FAR of 0.062% and FRR of 0%. Given the ability to obtain data for finger knuckles contactlessly, the ease in acquisition process, invariance of patterns to emotions and behavioral aspects, and a wide acceptability socially, there is potential in using this technology on a large scale undoubtedly. However, research is needed on improving the identification results for a widespread deployment.

Statistical features

Texture is a spatial property that is generally valid over a larger spatial neighborhood. In order to capture the spatial characteristics and dependence of the images, some statistical measures can be used in order to summarize the patterns indicated in the images. Using statistical methods typically yield good quality image descriptors with the requirement of a smaller memory footprint for maintaining the templates of the registered users. Various authors have worked on proposing statistical methods for feature extraction (Summary presented in Table 3).

Aoyama, Ito & Aoki (2014) proposed a FKP recognition algorithm based on block matching using phase correlation where the phase information was extracted from the 2D DFT (Discrete Fourier Transform) of the images. Good recognition results were obtained with a high main-to-side lobe ratio of correlation and an EER of 0.321%. Perumal & Ramachandran (2015) proposed a method to fuse the palm print and FKP of individuals using interest point based techniques including SIFT (Scale Invariant Feature Transform), SURF (Speeded-up Robust Features) and EMD (Emiprical Mode Decomposition) with even better results but it should be noted that EMD is an iterative algorithm and is not very suitable for real-time implementations. Veluchamy & Karlmarx (2016) made use of FKP and finger veins to design a biometric system in which repeated line tracking is performed followed by grid operation, yielding a set of features, which are used for classification using support vector machines (SVM). Although the overall results are good, there is a margin for improvement before the method can be deployed for commercial usage. Yang & Sun (2016) proposed a biometric system making use of palm print and palm veins, in which local binary patterns were used for feature extraction. The method achieves good results; however, a wider validation on a larger dataset is required for a thorough evaluation of the proposed method. Srivastava et al. (2016) performed the fusion of palm-phalages, palm print and dorsal hand vein using some statistical features (average absolute deviation, mean features and Gaussian membership) followed by classification using SVM, KNN (k-nearest neighbors) and random forest. However, the main aim of the authors was to introduce a new dataset for carrying out biometric research.

Chaudhary, Srivastava & Bhardwaj (2016) used palm print and dorsal hand vein for recognition, performing feature extraction using Gaussian membership function. Bhilare et al. (2018) made use of the images from hand vein based modalities by performing an ROI extraction, followed by the use of CS-LBP (Center Symmetric Local Binary Patterns) as texture features. They obtained very good results, concluding in their research that the palm vein and finger vein images yield better results rather than using the vein structure of the entire hand for recognition. Vishi & Mavroeidis (2018) made use of fingerprint and finger vein for performing recognition. The paper was aimed at a combination of score normalization and fusion techniques on the two aforementioned modalities, concluding that the hyperbolic tangent score normalization technique achieves the highest recognition rates. Yang et al. (2018b) aimed at improving the template protection in a multibiometric system in which fingerprint and finger veins were used. They made use of DFT based features, achieving very good recognition rates while ensuring the security of the templates. Korichi et al. (2018) proposed a multibiometric identification system with modalities composed of images obtained from both visible and near-infrared light. Feature extraction is performed using PCA (principal component analysis), with results obtained on various datasets, showing a high precision.

Yang et al. (2018a) proposed a Weber representation based feature extraction method for feature extraction from finger veins and dorsal veins. They performed feature level fusion and validated their results on several datasets, achieving very good results. Zhang et al. (2019) made use of multiple finger based modalities, followed by feature extraction using graph based methods. They achieved very good results, showing a high potential of employing graph theory for designing biometric recognition systems. Also, the results are validated on a large dataset, elucidating on the statistical significance of the achieved performance. Veluchamy & Karlmarx (2020) proposed a multibiometric system based on FKP and palm print in which they used HoG (histogram of oriented gradients) features. These are also part of the MPEG-7 standard and thus are used widely for multimedia application. Good recognition rates have been achieved; however, there is a margin of improvement in the overall recognition rates that could possibly be achieved using more specialized descriptors. Lv et al. (2020) made use of LBP based image descriptor for feature extraction from the fingerprint and finger vein images. Feature level fusion was employed to combine the fingerprint and finger vein patterns into a single image, followed by the use of contrast enhancement techniques, and later performing LBP giving good results. You & Wang (2019) discussed about the classical disadvantage of the fuzzy vault scheme in terms of potential attacks carried out by the attackers. They mitigated this problem by performing the fusion of fingerprint and finger vein templates, followed by the projection of feature points on a rectangular grid. The fuzzy vault scheme is later used for encoding and decoding purposes. Good results are obtained along with experimental evidence of the security of the proposed scheme.

Filter based features

Several feature extraction methods exist in the literature that are based on first filtering the images with a specific mask (filter) or a set of filters and then estimating the texture of the images based on models or statistics of the filter outputs. Typically, these methods result in a decomposition of the images giving a large amount of data based on the number of filters and their parameters (more descriptive with respect to some texture related characteristics such as edges, scales, angles, etc). Later, the decomposed information is summarized using some statistics, which enhance the description of the images with respect to the filter parameters. The literature is rich in terms of the use of filter based methods for texture feature extraction (Summary in Table 4).

Chin et al. (2014) predominantly focus on multibiometirc template protection. They have proposed a three-stage hybrid method. In order to obtain a fused template, fusion of fingerprint and palm print images is done at feature level, followed by applying random tile technique to obtain random features. These random fused features undergo discretisation, hence generating a secure template bit string. Khellat-Kihel et al. (2016) pointed out that the multimodal biometric system improves the accuracy significantly but on the other hand, they tend to have a larger memory footprint and result in longer execution times. Therefore, they proposed the extraction of features using Gabor filters, followed by feature selection using linear discriminant analysis (LDA) giving good recognition results. Gupta, Srivastava & Gupta (2016) proposed a hand geometry and vein pattern based method in which gradient based variational approach is used for the extraction of veins. Matching is performed using global approach in which Fourier Mellin transform is used, thus avoiding the issues such as non-uniform illumination, noise, etc. Hand geometry features are obtained using hand crafted features. The authors have carried out validation of methods on a large dataset with statistically significant results.

Verma & Dubey (2017) proposed a multimodal vein based recognition system in which an ROI (region of interest) is identified followed by vein enhancement using circular averaging filter and holoentropy thresholding. The results reported by Verma & Dubey (2017) are not very good; however, the feature extraction procedure is not precisely mentioned, and thus the quality of features employed cannot be fairly assessed. Bharathi & Sudhakar (2019) made use of hand vein based biometric modalities for performing recognition. Feature extraction was performed using Gabor filter and gradient based methods with matching performed using the Euclidean distance metric. Although good results are obtained, the authors have mentioned that using some other fusion techniques can improve the results. Furthermore, researching on a more relevant distance metric could potentially be useful for matching purposes. Kauba, Prommegger & Uhl (2019) proposed a contactless device to acquire images corresponding to hand-based biometric modalities, which make use of vein patterns for recognition. The authors collected a dataset for evaluation and also used various methods including Gabor filter, high frequency filtering, and interest point based methods for feature extraction. The proposed device achieves good results and exhibits potential for usage for recognition tasks. Jaswal & Poonia (2020) made use of palm print and finger knuckles to design an authentication system. They performed an ROI extraction from the respective images followed by line ordinal pattern based encoding of the images. Later, feature extraction is performed using criterion wavelet transform and feature selection is performed using linear discriminant analysis and search based methods yielding very good recognition rates. Li et al. (2021) proposed a joint discriminative feature learning framework in which the directional features are estimated using Gabor filters, which are later fed into an optimization framework for feature learning that maximizes the inter-class variation and minimizes the intra-class variation among samples. Finally, block-wise histograms of learned feature maps are used for recognition purposes, giving very good overall recognition accuracy of about 99.65%.

Deep features

Over the last couple of years, the Deep Convolutional Neural Networks have dominated significantly in terms of the extraction of features and for performing the classification tasks or solving recognition problems. This is because of their robust framework, having an incredible ability to learn from the training data and adapt the designed networks to solve complex problems. Even in biometric systems, the employment of deep learning has seen a significant surge and is producing very good results in comparison to the other methods that have been used previously (Summary in Table 5).

Zhong et al. (2018) proposed the use of DHN (deep hashing network) for palm print encoding into 128-bit codes, and BGM (biometric graph matching) to encode dorsal hand vein images into three discriminant features. Later, feature level fusion was used with very good recognition rates, with EER of upto 0%. Toygar, Babalola & Bitrim (2020) proposed a deep architecture with five hidden layers, each comprising convolutional, batch normalization and pooling layers to design a multibiometric system based on palm, dorsal, and wrist veins. The results when compared with several other methods including hand crafted features and Gabor filters, show very good results, elucidating on the potential of using deep learning methods for multimodal biometrics. Zhong, Shao & Du (2019) proposed a deep end-to-end trainable hashing network that takes an image at the input and outputs a binary code corresponding to the respective image. Matching can be performed by comparing the binary codes corresponding to the training images with the image given as input to the network. The method achieves very good results, showing promise in employing the neural networks based techniques for encoding the images. Chen et al. (2019) introduced a low cost personal identification system consisting of near infrared and visible LED (light emitting diodes). An adaptive feedback control was used to control the brightness of the diodes. The images acquired were preprocessed, with feature extraction performed using a deep scattering CNN, giving good recognition rates. Mehdi Cherrat, Alaoui & Bouzahir (2020) proposed a system for recognition using the CNN models in a multibiometric setting with a fusion of finger vein and fingerprint. Good recognition rates were obtained using the proposed strategy, with a conclusion that the use of preprocessing improves the recognition rates. Choudhury, Kumar & Laskar (2021) made use of index, middle, and ring fingernail plates for extracting biometric features from the images using three customized pretrained models: Alexnet, Resnet and Densenet. An adaptive fusion technique based on score and decision level is used for the purpose of fusion of features from dorsal hand followed by exhaustive experiments to assess the efficacy of the proposed technique, giving very good recognition results with a minimum average rate of 0.0097%.

Hand crafted features

Handcrafted features are typically those which are used with more traditional machine learning algorithms for performing classification tasks. More commonly, these features can be easily correlated to statistical features. In this specific article, we define the handcrafted features as those features that are obtained as a combination of statistical and physical properties of the images such as hand size, finger size, etc. Such combinations are typically obtained when at least one of the biometric traits used for recognition is hand geometry. Although the literature on such techniques is limited, a summary of relevant contributions made using such methods is presented in Table 6.

Sharma et al. (2015) performed identity verification using the shape and geometry of hands using the contour of the hands. The hand contour is initially aligned followed by the calculation of peaks and valleys, and the extraction of the finger feature points by calculating the Euclidean distance between the reference point and all the feature points. The proposed method shows promise with good recognition rates achieved over two datasets. However, their sizes are small and the methodology requires extensive validation over larger datasets for a statistically significant conclusion. Anitha & Rao (2016) made use of FKP and hand geometry to propose a multibiometric system. They performed ROI extraction from the pictures followed by the use of LBP as texture features for the finger knuckles and hand geometry features using hand crafted features. The process for the extraction of these features includes the identification of six points on the hand to extract the angle features and the aspect ratio of the palm, followed by the calculation of Euclidean distance between the template and the acquired image for matching. Experiments show that the best performance is achieved using a feature level fusion of the FKP and hand geometry features. Jaswal et al. (2019) laid down the idea of using multiple biometric traits for recognition using a single sensor. A device having the ability to capture the FKP and palm print was used for acquiring the image. Processing of the images was done by the extraction of an ROI followed by a transformation using texture code matrix. Hand registration was done by detecting the feature points (peaks and valleys) and also the detection of knuckle point followed by deep multiscale matching giving very good recognition rates. Gupta & Gupta (2018) proposed a system that captures slap fingerprints and hand dorsal image at the same time. Slap segmentation is performed by making use of the finger location and hand type. Matching of scores are generated by matching the slap fingerprints, palm dorsal vein and hand geometry that are fused for the purpose of authentication yielding good recognition results. Khodadoust et al. (2021) worked on the fusion of fingerprint, finger veins, and FKP using an experimental setup, which obtained the 3D reconstruction of the above-mentioned traits followed by maximum curvature based feature extraction. They obtained good overall identification results, with validation carried out on 66 users showing promise for the use of their method for authentication purposes. The most significant claimed advantage of the proposed method relates to the experimental protocol as the method relies on a contactless and hygienic way of acquiring multibiometric traits.

Now that we have analyzed the feature extraction techniques, which have been presented in the literature, it is important to note that there are a several points to compromise a biometric system. It is very important for a biometric system to be unsusceptible to attacks and loss of template by adversaries. To deal with the issues related to the security of a multibiometric template, we now analyze the existing work on multibiometric template protection.

Cancelable biometrics

Cancelable biometrics (CB) refer to distortion of biometric features that are intentional and systematically repeatable in nature to protect sensitive user-specific data (Ratha, Connell & Bolle, 2001). Cancelable biometric transforms are those that are used to transform the original biometric samples such that the resulting data is computationally hard to recover. When the user registers in the system, his biometric sample is transformed using a one-way transformation and saved in the database. This transformation is chosen from an identification word that is specific to the user. In the verification step, the query template is used to generate a transformed template that is compared with the saved template in the database followed by the verification process. The literature on unimodal cancelable biometric systems is very rich but there are inherent problems with such systems including intraclass variability, variation in data quality and a significant similarity in interclass samples. In contrast to such systems, the multimodal biometric systems combine the features of various biometric traits to generate the templates, which are more secure and thus, resistant to various threats and attacks. The main advantages offered by the multibiometric systems are greater security, accuracy, noise sensitivity, and resistance to spoof attacks. Table 8 presents a brief summary of the work done in the domain of cancelable multibiometircs.

Researchers have made several contributions on multibiometric template protection employing cancelable biometrics. Paul & Gavrilova (2012) proposed a method in which two-fold random selections are made from each biometric trait, followed by a feature level fusion. Random projection of each fold is obtained followed by PCA (principal component analysis) and later K-means clustering to generate the single template for individual biometrics. Later, LDA (linear discriminant analysis) is applied to further improve discriminability of the features. Final authentication is carried out using a classifier. Another variant of the technique proposed in Paul & Gavrilova (2013) makes use of Gram–Schmidt transformation instead of PCA, along with some other minor modifications in the pipeline. The authors have validated the cancelable property of the proposed method, while giving good authentication results in a multibiometric setting. Furthermore, the authors improved the results by proposing a methodology in which both Gram–Schmidt transformation and PCA were used followed by a rank level fusion for performing final authentication of the users (Paul & Gavrilova, 2014). Chin et al. (2014) proposed a three-stage hybird template protection scheme. They have performed the fusion of palm print and fingerprint on the feature level, followed by the use of random tiling technique to extract unique features. Finally, the fused random features undergo 2^N discretisation to produce the template bit string. The approach addresses the criterion for template protection with an improved EER as compared to unimodal biometric sytems, though it is slightly higher with reference to multimodal systems. Gomez-Barrero et al. (2017) made use of homomorphic probabilistic encryption to generate the biometric templates along with fusion at three different levels. A complexity analysis was also carried out to assess the feasibility of the proposed method for real-time implementation. Moreover, feature level fusion is also employed yielding more secure and better cancelable biometric features. Kaur & Khanna (2018) proposed a template transformation method named random distance method that yields privacy preserving, revocable and discriminative pseudo biometric identities, with about 50% reduced memory footprints. Yang et al. (2018b) proposed a multibiometric system in which the fingerprint based minutae features and finger vein features are extracted followed by their respective binary features, and then performing feature level fusion in three different ways. The method obtained secure biometric templates with good recognition results. Gomez-Barrero et al. (2018) showed the use of Bloom filter based protection schemes while elucidating that it is not a straightforward task. A statistical analysis of unprotected templates is carried out to estimate the main parameters of such schemes. Dwivedi & Dey (2019) proposed a method to obtain cancelable templates by using log-Gabor filters with phase quantization, followed by the generation of biometric codes. Score level fusion from multiple biometric templates are used for authentication yielding better results in comparison to unibiometric systems with better accuracy. Walia et al. (2020) proposed a method to obtain cancelable features using deep neural networks that are fused using adaptive graph based fusion method. The proposed method is used to obtain multimodal unified templates, which are empirically demonstrated to be robust to adversary attacks. Chang et al. (2020) proposed an authentication approach in which bit-wise encryption scheme is used to transform a biometric template into a secure template using a secret key, which is generated from another template. The scheme fully preserves the number of bit errors in the protected and original template, ensuring that the recognition performance is the same as that in the case of unprotected templates.

Biometric cryptosystems

Biometric cryptosystems (BCs) refer to designs that securely generate digital key from a biometric or bind a digital key to a biometric (Cavoukian & Stoianov, 2009). To overcome the shortcomings of traditional verification methods, which were based on password-based key-release, BCs bring about a considerable security benefit by offering biometric-dependant key-release since the biometrics have a strong link with the user’s identity (Uludag et al., 2004; Jain, Ross & Uludag, 2005; Rathgeb & Uhl, 2011). At the same time, combining biometrics with cryptography and extracting the keys is not that straightforward due to variations present in a biometric data. Most of the BCs require helper data that contains additional information about the biometric and is used to generate or retrieve a key (Jain, Nandakumar & Nagar, 2008; Ali & Khan, 2014). A helper data must not reveal significant information about original biometric templates. Table 9 presents a brief summary of the biometric cryptosystems.

Sutcu, Li & Memom (2007b) proposed the use of multibiometric features, followed by a secure sketch block, making it hard to extricate the original samples from the encrypted features. Nandakumar & Jain (2008) proposed a method to enable template protection using fuzzy vault framework. The authors claim to improve the recognition performance of the system along with enhanced security. Camlikaya, Kholmatov & Yanikoglu (2008) proposed a technique for the fusion of fingerprint template along with behavioral biometrics (voice samples). The algorithm enhanced the security of the biometric system by encoding the fingerprint features within the voice feature vector. The use of voice was motivated by using the property of spoken words used as a password to achieve the desirable cancelable property. Multiple biometric cryptosystems were proposed by Fu et al. (2009) out of which three were used for performing biometric fusion at the cryptographic level. The authors presented no experimental results; however, a detailed theoretical analysis of algorithms, comparison, and discussion were carried out. Nagar, Nandakumar & Jain (2011) provided a feature-level fusion method for both fuzzy vault and fuzzy commitment schemes that simultaneously secure the multiple templates of a user using a single secure sketch. Feature level fusion using multiple characteristics of a user proves to be significant in providing high privacy as compared to the single characteristic biometric systems, since only the fused feature vector is stored on the server database. Further, it requires less storage since only the combined feature vector is stored in the database server. However, it requires additional feature extraction and transformation tools for the heterogeneous features (variable formats based on distance, similarity, etc.). Another hybrid methodology to secure the biometric systems was proposed by Li et al. (2015) in which a combination of computational security and information security principles was implemented. Decision level fusion was done in the proposed cryptosystem for performing recognition. Kumar & Kumar (2015) proposed a multibiometric system based on cell array. Encoding and hash code computation was done using Bose Chaudhuri Hocquenghem (BCH) on the biometric modalities. The data is scattered across the two cells such that the first cell stores the hash code and the second cell stores the key. Moreover, fusion was performed at both decision and feature levels out of which the former shows better results in a multibiometric cryptosystem setting. You & Wang (2019) proposed a novel fuzzy vault scheme, which effectively protects the multibiometric template against location attack, brute force attack, and correlation attack. They performed fusion of fingerprint and finger vein templates. Feature point fusion encoding is done through grid projection, and fusion encodings are applied to construct the fuzzy vault. Chang et al. (2021) proposed BIOFUSE in which fuzzy commitment and fuzzy vault are combined using an encryption scheme. The system makes it difficult for an attacker to gain unauthorized access to the system without doing an impersonation of all the biometric traits at the same instant. The experiments have shown very good recognition rates with a high security. Evangelin & Fred (2021) used a visual shadow creation process to create multiple shadows of one image followed by encoding and decoding using elliptic curve cryptography. Although a very secure model is obtained, the implementation time of the model was significantly expanded. Asthana, Walia & Gupta (2021) made use of a key binding mechanism to generate a secret key using the biometric data of the user, leading to the proposition of a biocrypto system. Novel objective functions are proposed to create the helper data. The local minima of the objective function are taken as anchor points to retrieve the secret key and perform recognition leading to about 98% success rate in recognition even in the presence of limited noise in biometric data.

Challenges

The limitations faced by various researchers in the implementation of biometric systems are listed as under:

• Aging/Alteration: It is well known that even when the biometric traits do not suffer any natural changes over a period of time, they are subject to changes due to trauma or physical damage due to cuts, different skin conditions, or other unforeseen events. Even when there is no medical/physical condition, which is liable to cause any damage to the biometrics, the traits change over a period of time (Lanitis, 2010; Trokielewicz, Czajka & Maciejewicz, 2018). This is a challenging problem and as such there is no remedial solution available for such problems other than making use of the biometric traits that are less sensitive to such alterations. Among the hand biometric systems, there are several modalities which exhibit the information not from the skin but the sub-skin structures (veins) that are captured using the IR camera. A mild or superficial skin condition does not affect the vasculature, hence the vein based multibiometric systems have the ability to cater for most of the problems occurring due to trauma on the skin such as cuts. Aging and some other chronic conditions such as hypertension, diabetes, etc., affect the vein biometrics (changing in the diameter of the veins) and thus can impact on the recognition performance of the biometric systems (Xie et al., 2017). However, since these changes do not take place overnight, there are ways to mitigate these problems.

• Operational problems: Operational problems refer to the various problems that have the ability to affect the performance of multibiometric systems. These problems can result from various factors such as environment and the methodology of acquisition of the data. The good thing is that most of these operational problems can be mitigated by the acquisition of data multiple times, until a good sample is captured from the acquisition device. For example, if there is excessive moisture in the fingers, the sample captured from the device may be having specular reflection (Auksorius & Boccara, 2017). Such problems can be mitigated if the fingers/hands are cleaned for any moisture before application to the sensor. Apart from environmental conditions, even when the proper methodologies for acquisition of data have been followed, it is possible to face some issues such as alignment. If the biometrics are not properly aligned according to the templates and are, for example, captured at different angles than the templates, then it is possible to handle such problems with one of the modules, such as feature extraction. Therefore, it should be noted that the features should be extracted such that they are invariant to some underlying imaging conditions such as illumination and rotation at the very least.

• User errors: The orientation and shifting of the fingers during registration and authentication process significantly affects the performance of the biometric systems based on fingers (Liu et al., 2014; Peng et al., 2012; Dong et al., 2014). Furthermore, during the imaging process, any movement of the fingers or hands causes irregular illumination (Song et al., 2011). As a result, different segments of the fingers/hands get different amount of light absorption and as a result, the quality of the acquired images is not adequate for performing recognition.

• Biometric finger features: Another important factor which plays its role is finger features. Studies have shown that in addition to the varying thickness of fingers, certain factors related to finger skin affect the image capturing process, such as skin pigmentation, thickness, hair, etc. (Gupta & Gupta, 2015). Furthermore, studies have shown that the varying thickness of finger skin results in an unequal distribution of light passing through the skin, hampering in the collection of high quality vein patterns.

• Complexity of fusion: It is clear that the use of multibiometrics leads to a better recognition and can help in increasing the security of the systems. However, with the use of more than one trait, there are four possibilities of performing the process of fusion on the biometric templates. This leads to critical decision making in any system as it could define the performance as well as the security characteristics of the systems (Dinca & Hancke, 2017; Zhong, Du & Zhong, 2019). Moreover, biometrics is a domain that requires real-time performance and fusion will typically require the system to process a large amount of data in comparison to a unibiometric system (which does not require fusion). The recognition rates by adopting various methods of fusion could vary based on the platforms (e.g., mobile, wearable devices, etc.) and architectures (biometric traits, feature extraction methods). Therefore, an exploratory study of the performance yielded by different levels of fusion using standardized platforms and architectures to analyze the best fusion methods is a significant challenge.

• Security of multibiometric templates: The security aspect in biometric systems has recently gained a significant traction. There are several techniques that have been explored well including different fusion schemes, convolutional methods using different types of filters, methods for generating secure sketches and fuzzy vault constructs, encryption schemes, etc. In this context, the literature is very rich in regard to the unibiometric systems. The literature on the security of multibiometric systems is very limited and thus there is a lot of scope of work available in this area. The most widely explored methods used for multibiometric systems are derived from transformation based methods and fusion techniques. Generally, the fusion techniques are used in conjunction with a variety of other methods used for inducing security in multibiometric systems. This is because, there are different types of fusion that take place at different stages in the pipeline of multibiometric systems. However, it should be noted that the feature based fusion is the most common type of fusion technique, which is used for enhancing security, as it gives a richer set of features, generally yielding better evaluation metrics for multibiometric systems. It would not be wrong to say that most of the proposed techniques are focused on the use of hand crafted features, along with a mixture of fusion techniques, with a limited focus on representation learning based algorithms.

• Lack of standard performance measures: There are several measures that have been used to assess the performance of a biometric system. Due to the lack of a single measure to quantify the system, it is very hard to make a comparison of different methods that have been published in the literature (Ryu et al., 2021; Kumar, Prasad & Raju, 2020; Manisha & Kumar, 2020). There is a need to work on a unified measure that is widely adopted by the researchers to evaluate different authentication systems, and making the literature standardized and valid for performing direct comparisons.

Future Directions for Improving Biometric Systems

The main future directions of work for the implementation of biometric systems are summarized as follows:

• Multi-sample registration: One method that is typically used to solve the problem of variations in the data acquisition for a single person is doing a multi-sample registration, in which multiple samples are captured for a specific trait. In this way, the variations of a single sample are captured and the machine learning methods are appropriately trained to capture this intra-sample variation.

• Rotation and illumination invariant descriptors: As discussed previously, one problem that is faced during the sample acquisition is capturing the samples at varying angles. It is practically not possible to always capture the data of the hand biometrics exactly aligned according to the available templates. This could be due to both the acquisition of the template or the sample. An interesting way to mitigate this problem is to work on image descriptors, which are invariant to the image capturing conditions (Riaz et al., 2013). Specifically, if the descriptors are rotation and illumination invariant, this problem can be effectively addressed. However, it should be noted that if the descriptors are illumination invariant, it would not be possible to integrate the information about soft biometrics (such as color of hand) within the template and the respective features.

• Incremental machine learning (IML): A very interesting area that has recently gained traction is IML. As discussed, the biometrics are bound to changes over a period of time. Some biometric traits undergo more transformations as compared to the other. If somehow these changes are properly recorded, they can be effectively handled while still yielding higher recognition rates. There are two main methods to solve these issues (Mehrotra et al., 2016): one way is to keep updating the templates as the user is authenticated. Another way is to make the ML algorithm learn new parameters while a new sample is presented for a user. Both these methods are effectively used for incorporating adaptivity to some extent.

• Fusion based adaptation: The process of adaptation in biometrics is not only limtied to templates, but it can also be extended to the fusion level. For instance, when performing fusion of multibiometrics, it is possible to perform fusion on the decision level by giving more weightage to the traits which are more stable over a period of time as compared to those which are not. In this context, there are several examples in the literature with regard to adaptive score weighting (Assaad & Serpen, 2015; Sim et al., 2014), score normalization (Khalifa, Gazzah & BenAmara, 2013), adaptive feature weighting (Huang et al., 2015; Xu & Lu, 2015), etc.

• Soft biometrics: This paper mainly discusses the literature on hard biometrics in which the physiological biometric traits are used to authenticate the users based on their mathematical modeling. Lately; however, soft biometrics have been gaining attraction since they have the ability to complement the biometric systems with their decision making. The idea behind this concept is that the biometric systems make decision about a specific user based on some characteristics such has skin colour, eye color, height, weight, beard, etc. Interestingly, hand based biometric systems take this liberty to use these soft biometrics to perform recognition. This is because, there are at least two very important characteristics that can be used for recognition in hand based multibiometric systems, i.e., skin color and hair. The skin color can be used as a property of the individuals, whereas the presence of hair on the hands or the texture of hands from dorsal view can indicate the gender of the individual. Also, previous studies have shown that the hand measurements, hand length, hand breadth, palm length, palm breadth, etc., can be correlated to the gender of an individual (Rastogi, Murali & Rastogi, 2014). Soft biometrics can be used for recognition in a mixed authentication setting where these are used in conjunction with the hard biometrics for authentication.

• Multibiometric template protection schemes: Although the work on multibiometric template protection is limited, this is bound to change in the future given that there is an increasing interest of researchers in extending such algorithms with their variants that yield better results. Moreover, the focus of researchers currently is on using the deep features/algorithms in various domains rather than focusing on the classical classification algorithms (Khan et al., 2019). A notable recent contribution related to cancelable biometrics using deep features is the technique proposed in (Walia et al., 2020), where the authors have used a modified of Resnet model generating deep representation of biometric traits, followed by graph based fusion for generating a unified template. Given that cancelable biometrics make the biometric templates non-recoverable offering a high security, but the recognition rate is expected to be compromised. The method offers better results in comparison to some other methods considered in this paper; however, there is a scope for improvement as can be seen by the performance where the EER of the proposed method is 4.34%. Another potential direction for future research is the use of hybrid methods for template protection, which offers combined benefits of several methods. It should be noted that this requires caution due to the fact that the performance of authentication should still stay real-time and the employment of multiple computationally complex algorithms may require powerful computing resources.