mdedup

systemds.operator.algorithm.mdedup(X: Frame, LHSfeatures: Matrix, LHSthreshold: Matrix, RHSfeatures: Matrix, RHSthreshold: Matrix, verbose: bool)

Implements builtin for deduplication using matching dependencies (e.g. Street 0.95, City 0.90 -> ZIP 1.0) and Jaccard distance.

Parameters:

• X – Input Frame X

• LHSfeatures – A matrix 1xd with numbers of columns for MDs (e.g. Street 0.95, City 0.90 -> ZIP 1.0)

• LHSthreshold – A matrix 1xd with threshold values in interval [0, 1] for MDs

• RHSfeatures – A matrix 1xd with numbers of columns for MDs

• RHSthreshold – A matrix 1xd with threshold values in interval [0, 1] for MDs

• verbose – To print the output

Returns:

Matrix nx1 of duplicates