Search before asking
Description
I'm attempting to do duplicate analysis by hashing several fields and storing the hash. If the hash already exists then the item is determined as a duplicate.
Using XXHash_64 results in a hash collision chance of ~93% with 10 billion items (based on this calculator https://kevingal.com/apps/collision.html).
I could use another hash function such as MD5 or SHA. In the case of MD5 it's much slower and would effect performance. In the case of SHA I would need to store the hash as a string as the largest integer type is 128 bits, which prevents me from using the hash column as a key.
From what I can see implementing a XXHASH_128 function would be the best option, though I might be overlooking something as I've only started using Doris. Any suggestions are welcome, thanks.
Solution
Implement XXHASH_128.
I'm willing to submit a PR but I'm not sure where to start, any guidance would be appreciated.
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
I'm attempting to do duplicate analysis by hashing several fields and storing the hash. If the hash already exists then the item is determined as a duplicate.
Using XXHash_64 results in a hash collision chance of ~93% with 10 billion items (based on this calculator https://kevingal.com/apps/collision.html).
I could use another hash function such as MD5 or SHA. In the case of MD5 it's much slower and would effect performance. In the case of SHA I would need to store the hash as a string as the largest integer type is 128 bits, which prevents me from using the hash column as a key.
From what I can see implementing a XXHASH_128 function would be the best option, though I might be overlooking something as I've only started using Doris. Any suggestions are welcome, thanks.
Solution
Implement XXHASH_128.
I'm willing to submit a PR but I'm not sure where to start, any guidance would be appreciated.
Are you willing to submit PR?
Code of Conduct