Using hashes -- pb's vs xxhash and murmur3

[jassing]

Don't really know why murmor is faster than FNV

Neither do I.

[wilbert]

Murmur is faster because fnv handles only one byte at a time.
What I don't like about Murmur3 is that the x32-128 bit and x64-128 bit versions create different hashes.
xxHash produces the same hash on both x32 and x64 but it's only a 32 bit hash and I don't know if that's enough these days.
So maybe Murmur3 is best in most cases. It should be pretty fast on large files.

[jassing]

Murmur is faster because fnv handles only one byte at a time.
What I don't like about Murmur3 is that the x32-128 bit and x64-128 bit versions create different hashes.

You can run the x86 version of the hash on x64 OS -- same hash then for both. Only slight speed hit.

[wilbert]

You can run the x86 version of the hash on x64 OS -- same hash then for both. Only slight speed hit.

Yes, you are right about that. There shouldn't be a big difference running the x86 version on x64 compared to running that version on x86.
But the speed difference with the version designed for x64 is so big.
The wiki page http://code.google.com/p/smhasher/wiki/MurmurHash3 mentions a speed on their test system of 2684 mb/sec for the x86 version and 5058 mb/sec for the x64 version.
That's a huge difference.
I don't know what version would be most suitable if I would try to convert one to PB.

[jassing]

I don't know what version would be most suitable if I would try to convert one to PB.

Use the right tool for the job. windows static libs will work regardless of pb version (unlike "userlibs"); You have the source if needed, so why convert to PB, when the C code is already very fast and optimized? Just to say that it is all "pb"? I doubt you'll gain anything...

I had hoped for some feedback on the two hashes, not a "use this one instead" etc; the speed is unbeatable so far, so I've run with it...

[wilbert]

I had hoped for some feedback on the two hashes, not a "use this one instead" etc; the speed is unbeatable so far, so I've run with it...

In that case ...
If I had to choose between xxHash and MurmurHash3, I would go with MurmurHash3.
It's much more used and as far as I know turned out to be reliable, you have the choice between a 32 bit hash or a 128 bit hash, it is public domain code and still very fast.

[jassing]

I had hoped for some feedback on the two hashes, not a "use this one instead" etc; the speed is unbeatable so far, so I've run with it...

In that case ...
If I had to choose between xxHash and MurmurHash3, I would go with MurmurHash3.
It's much more used and as far as I know turned out to be reliable, you have the choice between a 32 bit hash or a 128 bit hash, it is public domain code and still very fast.

the thought was to use murmur3 on large file data sets and xxhash for transmission block testing
So -- transfer "1 block" <verify xxhash> "block #2" <xxhash> etc.
then once file is transferred & patched, re-verify using murmur3; if invalid, repeat.
Satellite modems are not 100% reliable , just like old telcom modems...

I hadn't considered just using the 32 bit murmur -- that definitely makes it less "complicated" in dealing with only one hash library.

Thanks for the thoughts.