FDup: a framework for general-purpose and efficient entity deduplication of record collections

Collection import

FDup operates over a set of flat records, whose structure consists of a set of labels (custom name) and values (extracted by the original record). Therefore, the first step of the workflow consists in mapping the target collection of records, which may not be flat, onto an FDup flat collection of records with labels [l₁, …, l_k], which will be used as the template for the configuration of the deduplication steps.

In this article, the experiments are conducted over a collection of bibliographic records for scientific publications as provided by the OpenAIRE Research Graph, featuring the following flat structure:

• PIDs: a sequence of values denoting persistent identifiers of the records, i.e., unique identifiers; each record may have more than one PID, released by different agencies at the moment of depositing the article, such as DOIs in Crossref, ArXiv identifiers, PubMed identifiers, etc.;

• title: the title of the article;

• abstract: the abstract of the article;

• authors: the list of authors and contributors of the article, provided as a list of strings typicaly following different formats, e.g., “J. Smith”, “Smith, J.”, “Smith, John H.”, “J.H. Smith”;

• date: the date of publication of the article, harmonized to a common format dd/mm/yyyy;

• venue: the venue of publication, such as the conference or a journal, typically comes as a string of free text.

Candidate identification

The ideal similarity matching process, where every pair of records is confronted to yield in an equivalence score, features quadratic complexity and known performance issues. Two common solutions are the techniques of clustering/blocking and sliding window. Both methods apply heuristics to identify, prior to record matching, a selection of record pairs in the collection that are candidates for equivalence.

Clustering functions are of the form clusteringKey ([l₁, …, l_k]) and are applied to all records in the initial collection to produce one or more keys per record. Functions should be smart enough to ensure that potentially equivalent records likely return the same key. Records are then grouped by key into blocks, and pair-wise comparisons are executed within such blocks. For the scientific publication records above, a reasonable clustering function is one that generates n-grams (fragments of n characters of a string) from the title words or one that generates the PIDs (especially the DOI) when these are available.

Sliding window methods introduce a further optimization of the number of comparisons. Records in one block are sorted by a key (typically obtained from a value of the record’s field) (orderField) to obtain an array that is visited from the first element to the last. At every iteration, the pivot record is matched with the subsequent ones in a limited K interval (window). The key used for the sorting should be generated in such a way that similar records are likely close in the ordering, possibly within the window range K.

FDup offers the possibility to easily configure every single step previously described via single configuration profile, including: a pre-defined and extendable set of clusteringKey functions, the orderField, and the length K of the sliding window.

Duplicate identification: T-match function

Duplicate identification consists of a similarity function that matches the equivalence conditions between two records in a block. As such, it is defined by matching the values of the records fields in such a way domain-specific conditions of equivalence are met.

For scientific publication records, OpenAIRE defines two records as equivalent when they describe the same scientific work, hence one object for the purpose of measuring impact. For example, different depositions of the same article in distinct repositories (e.g., ArXiv.org, Zenodo.org) are to be considered equivalent, i.e., cannot be counted as two independent scientific results. For the same reason, two different versions of a document are to be considered different, as they denote different efforts. For example, version 1 and version 2 of a project deliverable.

In real case scenarios, like the one of OpenAIRE, where metadata is harvested from highly heterogeneous data sources, many of the field values in the records are not and cannot be sufficiently harmonized to the degree of uniformity required to make them reliable in the process of equivalence check. This is the case for abstract, for which establishing a distance and weight for it is not trivial, and venue whose values can hardly be harmonized into comparable values. Moreover, the date field cannot be used as a discriminator, as different versions of the article, published at different times, may indeed be regarded as equivalent.

Record equivalence is assessed by relying on three fields, PID, title, and authors, when matching the following considerations:

• Equivalence by identity: when two records have one PID in common, they are equivalent;

• Equivalence by value: when PIDs are not matching, the records may still be equivalent (e.g., deposited in different repositories), hence a field value equivalence matching is required.

While equivalence by identity is rather straightforward, equivalence by value requires context-driven rules, which in the case of publication match in OpenAIRE are: (i) a rather high title equivalence confidence of 99%, as indeed typos may occur; (ii) ensuring that the 0,01% of difference in the titles is not due to a number or Roman number denoting different versions of the publication; and, (iii) making sure the records have corresponding authors, by checking their names.

The majority of deduplication frameworks in the literature encodes record similarity match conditions via a similarity function of the form: ${\displaystyle f\left(\left[v_1,\dots ,v_k\right],\left[v_1^{\prime },\dots ,v_k^{\prime }\right]\right)=\sum _{i:0\dots k}{f}_i\left(v_i,v_i^{\prime }\right)\times w_i}$

where the v_i’s are the values of field l_i, $f_i\left(v_i,v_i^{\prime }\right)$ are comparators, functions measuring the “distance” of v_i and $v_i^{\prime }$ for the field l_i, and w_i’s are the weights assigned to the comparators f_i’s, such that ∑_i:0…kw_i = 1.

As a result, f returns a value in a given range, e.g., [0…1], scoring the distance between two records. The records are regarded as equivalent if the distance measure is greater than a given threshold.

For the example above, the similarity function PublicationWeightedMatch, created using the GDup framework in OpenAIRE, encodes both equivalence by identity and by value as follows: ${\displaystyle \begin{array}{cc}PublicationWeightedMatch\left(r,r^{\prime }\right)=\hfill & jsonListMatch\left(r.PIDs,r^{\prime }.PIDs\right)\times 0.5\hfill \\ \hfill & +TitleVersionMatch\left(r.title,r^{\prime }.title\right)\times 0.1\hfill \\ \hfill & +AuthorsMatch\left(r.authors,r^{\prime }.authors\right)\times 0.2\hfill \\ \hfill & +LevenshteinTitle\left(r.title,r^{\prime }.title\right)\times 0.2\hfill \end{array}}$

where jsonListMatch, applied to the field PID, returns 1 if there is at least one PID in common in the two records; TitleVersionMatch, applied to the titles, returns 1 if the two titles contain identical numbers or Roman numbers; LevenshteinTitle returns 1 if the two (normalized) titles have a Levenshtein distance greater than 90%, and AuthorsMatch performs a “smart” matching of two lists of author name strings and returns 1 if they are 90% similar (the minimal equivalence threshold is computed over a manually validated ground truth of equivalent records). All comparators return 0 if their condition is not met. The minimal threshold for two records to be equivalent is 0.5, the threshold that can be reached by jsonListMatch alone or by combining the positive results of the three functions TitleVersionMatch, AuthorsMatch, and LevenshteinTitle.

All f_i’s in PublicationWeightedMatch are computed at the same time and averagely require a constant execution time, despite the successful or unsuccessful match that those may feature. Motivated by such observation, FDup introduces a similarity match function T-match that returns an equivalence match exploiting a decision tree, nesting the comparator functions. Each tree node verifies a condition, which can be the result of combining one or more comparators, and introduces a positive (MATCH) or negative (NO_MATCH) exit strategy. If the exit strategy is not fired, T-match heads to the next node. An early exit skips the full traversal of the tree and can turn the result into a MATCH, i.e., a simRel relationship between the two records is drawn, or into a NO_MATCH, i.e., no similarity relationship is drawn.

A T-match decision is formed by a tree of named nodes with outgoing edges. The core elements of a T-match node are the aggregation function, the list of comparators, and a threshold value. The aggregation function collects the output of the comparators and delivers an “aggregated” result based on one of the following functions: maximum, minimum, average, and weighted mean. Each comparator in a node accepts two values of the input records for a given field and returns a value in the range 0…1. Notably, different comparators in an aggregation can refer to different fields, giving high degree of customization to end-users (in the following, one node will encode a weighted mean function as the one described above). The execution of a T-match node must end with a decision, which may be:

• positive, i.e., the result of the aggregation function is greater or equal than the threshold value;

• negative, i.e., the result of the aggregation function is lower to the threshold;

• undefined, i.e., one of the comparators cannot be computed (e.g., absence of values); a node also bears a flag ignoreUndefined that ignores the undefined edge even if one of the values is absent.

For each decision, the node provides the name of the next node to be executed. By default, T-match provides two nodes MATCH and NO_MATCH to be used to force a successful or unsuccessful early exit from the tree.

The example in Fig. 2 shows the function PublicationTreeMatch, which uses the same comparators but exploiting a T-match decision tree. The individual matches are lined up by introducing MATCH conditions early in the process, i.e., equivalence by identity via PIDMatch, and then ordering NO_MATCH conditions by ascendant execution time, i.e., equivalence by value via versionMatch, titleMatch, and authorsMatch.

Independently of the domain, smart identification of exit strategies becomes a means for developers to reduce the overall deduplication time. Moreover, T-match allows for the definition of multiple paths, hence the simultaneous application of alternative similarity match strategies in one single function. The experiments described in later sections will show that when the number of records is very large, T-match significantly improves the overall performance of the deduplication process.

Duplicates grouping

The outcome of the duplicate identification phase is the graph resulting from combining the input collection of records with the set of simRel relationships between them. The duplicate grouping phase first finds the sets of equivalent records by calculating the connected components in the graph via the transitive closure of the simRel relationships; for example, A simRel B and B simRel C identify the group A, B, C. Secondly, it generates a new graph, where the groups of equivalent records are all linked with a mergeRel relationship to a representative record, created by the process to identify the groups; for example, the group of nodes A, B, C will deliver the graph of four nodes A, B, C, R with the relationships A mergeRel R, B mergeRel R, C mergeRel R.

The Configuration file

The FDup’s configuration file is expressed in JSON format and consists in two different sections: pace, which defines the T-match function parameters; and workflow, which defines the deduplication workflow parameters.

Pace section.

The section specifies the configuration for the pair-wise comparisons, including the data model, the record pairs’ blacklist, the synonyms, and the decision tree of the T-match function.

FDup operates on a collection of flat records with the same structure. As the original collection of JSON records may not be flat, FDup introduces the mechanism of the data model. The data model subsection of the configuration drives the transformation of the original JSON records onto a flat record with labels and values as depicted in Fig. 4. The mapping is defined by means of JSON paths, whose result is implicitly assigned to a given field of the FDup’s record data model. Additional parameters such as length and size can be used to limit the value to take from the original JSON entity.

The clustering subsection includes the list of blocking functions (more than one can be used) to be used for the key extraction. Each clustering function specifies the list of record fields to which the function should be applied. The user can also specify parameters for the clustering function such as key length, maximum number of keys to extract and other configurations. The structure of a clustering function is depicted in Table 1.

The blacklist is a list of field values to be excluded from the computation. The mechanism is useful to exclude false positives from subsequent rounds of deduplication. It is expressed as a map in which the key is the field name while the value is the list of strings to exclude. When the process is running, entities with one of those values for the specified field will be ruled out of the comparisons.

The synonyms are lists of equivalent terms. They are typically used to encode semantic equivalence across different vocabularies (e.g., replacing terms with a common code). For example, synonyms are exploited to address translations of terms across different languages and therefore capture their equivalence.

The decisionTree section sets the configuration of T-match’s pair-wise comparison algorithm. T-match’s tree can be shaped up by creating different nodes, each composed by one or more comparators. Each comparator acts on a pair of records and over a given pair of fields, to return a number that reflects the similarity of two fields. The results of the comparators in one node are then aggregated to return an overall similarity score. The user can define the aggregation function to be used (i.e., weighted mean, average, maximum, minimum, etc.). The structure of a comparator and a tree node are depicted in Table 2 and Table 3.

Table 1:

Definition of a clustering function.

Field	Description
Name	The name of the clustering function
Fields	The list of fields to which the clustering function should be applied
Params	The list of parameters to configure the clustering function. Every parameter has a name and can assume a number as value. Those parameters are accessible from the clustering function.

DOI: 10.7717/peerjcs.1058/table-1

Table 2:

Definition of the comparator.

Field	Description
Field	The field to compare (it must be defined in the model)
Comparator	The comparator to use for the comparison
Weight	The weight of the score for the comparator. The value is used when the comparator’s scores are aggregated to give a weight to the comparator.
countIfUndefined	Boolean value that specifies if the score should be considered in the aggregation also when the result of the comparison is undefined (i.e., missing field)
Params	The list of parameters for the comparator. Every parameter has a name and can assume a string as value. Those parameters are accessible from the comparator.

DOI: 10.7717/peerjcs.1058/table-2

Table 3:

Definition of the tree node.

Field	Description
Fields	The list of comparators contributing to the computation of the final score
Threshold	The threshold for the final score of the node. If the final score is greater than the threshold, the execution will follow the positive edge; otherwise, it will follow the negative edge. The execution will follow the undefined edge when the ignoreUndefined is not enabled and one of the comparators returned an undefined result.
Aggregation	The aggregation function to use for the computation of the final score (e.g., maximum, minimum, average, weighted mean, ecc.)
Positive	The name of the next node to compute if the final score of this node is greater than or equal to the threshold.
Negative	The name of the next node to compute if the final score of this node is lower than the threshold.
Undefined	The name of the next node to compute if the result of the node is undefined.
ignoreUndefined	Boolean value that specifies if the undefined tree edge should be ignored or not.

DOI: 10.7717/peerjcs.1058/table-3

Pace_Core module: Block Processor and T-match Processor

The BlockProcessor is the Java class that provides functionalities to process a block (e.g., entity sorting), it implements the sliding window mechanism and it invokes the T-match Processor to performing the record pair-wise comparisons. The core of this class is the function that implements the queue management when blocks of records are sorted. Records in a block are queued in alphabetical order (of the orderField) and subsequently the queue is processed and pair-wise comparisons are computed.

The T-match Processor is the Java class that navigates the decision tree by calculating the result for every node according to the configuration provided in the Configuration file. Such class is invoked by the BlockProcessor and basically invokes each comparator involved in the node in order to collect and aggregate their scores (as specified by the user). At the end of the aggregation, a threshold is applied to the final score and the next tree node to process is chosen.

For the example in Fig. 2, when the T-match Processor is computing the score of the second node of the tree, it firstly calculates the scores for the two comparators (i.e., titleVersionMatch and sizeMatch) and then aggregates them with an AND operation (i.e., both conditions must be satisfied).

Pace_core module: libraries

As stated before, FDup offers a set of predefined and extendable libraries of comparators and clustering functions. They are implemented as Java classes inside the framework package.

The user can customize the framework by implementing new components depending on its needs. In order to implement a new comparator, it is sufficient to implement a Java class that extends the proper interface defined in the package.

The Dedup_Workflow

This module implements the workflow depicted in Fig. 1 by building on the Apache Spark Framework, which allows to define and configure applications with high performance for both batch and streaming data. The framework can be used with different programming languages (Java and Scala in FDup) and offers over 80 high-level operators facilitating the realization of parallel applications.

The deduplication workflow is implemented as an Oozie workflow that incapsulates jobs executing the three steps depicted in Fig. 3, to compute: (i) the similarity relations (SparkCreateSimRels), (ii) the merge relations (SparkCreateMergeRels), and (iii) the groups of duplicates and the related representative objects (SparkCreateDedupEntity). More specifically:

• SparkCreateSimRels: uses classes in the Pace_Core module to divide entities into blocks (clusters) and subsequently computes simRels according to the Configuration file settings for T-match;

• SparkCreateMergeRels: uses GraphX library to process the simRels and close meshes they form; for each connected component, a master record ID is chosen and mergeRels relationships are drawn between the master record and the connected records;

• SparkCreateDedupEntity: uses mergeRels to group connected records and create the representative objects.