class freediscovery.near_duplicates.SimhashNearDuplicates(hash_func='murmurhash3_int_u32', hash_func_nbytes=32)[source]

Near duplicates detection using the simhash algorithm.

A classical near-duplicates detection involves comparing all pairs of samples in the collection. For a collection of size N, this is typically an O(N^2) operation. Simhash algorithm allows to retrieve near duplicates with a significantly better computational scaling.


this estimator requires the `simhash-py <>`_Python package to be installed.

  • hash_func (str or function, default='murmurhash3_int_u32') – the hashing function used to hash documents. Possibles values are “murmurhash3_int_u32” or a custom function.
  • hash_func_nbytes (int, default=64) – expected size of the hash produced by hash_func


[Charikar2002]Charikar, M. S. (2002, May). Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing (pp. 380-388). ACM.
fit(X, y=None)[source]
Parameters:X ({array, sparse matrix}, shape (n_samples, n_features)) – List of n_features-dimensional data points. Each row corresponds to a single data point.
Returns:self – Returns self.
Return type:object

Get document index by hash

Parameters:shash (uint64) – a simhash value
Returns:index – a document index
Return type:int

Get parameters for this estimator.

Parameters:deep (boolean, optional) – If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:params – Parameter names mapped to their values.
Return type:mapping of string to any
query(distance=2, blocks='auto')[source]

Find all the nearests neighbours for the dataset

  • distance (int, default=2) – Maximum number of differnet bits in the simhash
  • blocks (int or 'auto', default='auto') – number of blocks into which the simhash is split when searching for duplicates, see

  • simhash (array) – the simhash value for all documents in the collection
  • cluster_id (array) – the exact duplicates (documents with the same simhash) are grouped by in cluster_id
  • dup_pairs (list) – list of tuples for the near-duplicates


Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as pipelines). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Return type:self