Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Robust and efficient fuzzy match for online data cleaning

Surajit Chaudhuri, Kris Ganjam, Venkatesh Ganti, and Rajeev Motwani

Abstract

To ensure high data quality, data warehouses must validate and cleanse incoming data tuples from external sources. In many situations, clean tuples must match acceptable tuples in reference tables. For example, product name and description fields in a sales record from a distributor must match the pre-recorded name and description fields in a product reference relation. A significant challenge in such a scenario is to implement an efficient and accurate fuzzy match operation that can effectively clean an incoming tuple if it fails to match exactly with any tuple in the reference relation. In this paper, we propose a new similarity function which overcomes limitations of commonly used similarity functions, and develop an efficient fuzzy match algorithm. We demonstrate the effectiveness of our techniques by evaluating them on real datasets.

Details

Publication typeArticle
Published inSIGMOD
PublisherACM – Association for Computing Machinery
> Publications > Robust and efficient fuzzy match for online data cleaning