Ping Li, Arnd Christian König, and Wenhao Gui
6 December 2010
Computing two-way and multi-way set similarities is a fundamental problem. This study focuses on estimating 3-way resemblance (Jaccard similarity) using b-bit minwise hashing. While traditional minwise hashing methods store each hashed value using 64 bits, b-bit minwise hashing only stores the lowest b bits (where b <= 2 for 3-way). The extension to 3-way similarity from the prior work on 2-way similarity is technically non-trivial. We develop the precise estimator which is accurate and very complicated; and we recommend a much simplified estimator suitable for sparse data. Our analysis shows that b-bit minwise hashing can normally achieve a 10 to 25-fold improvement in the storage space required for a given estimator accuracy of the 3-way resemblance.
|Published in||Twenty-Fourth Annual Conference on Neural Information Processing Systems (NIPS)|
Ping Li and Arnd Christian König. b-Bit Minwise Hashing, Association for Computing Machinery, Inc., 26 April 2010.