Preserving data Comparison of main checksums Name MB/s on intel core 2 Cycles Per Byte Adler32 920 1.9 MD5 255 6.8 SHA-1 153 11.4 SHA-256 111 15.8 SHA-512 99 17.7 cksum block 16/43 S.Ponce-CERN
Preserving data 16 / 43 S. Ponce - CERN risks consistency safety c/c cksum block Comparison of main checksums Name MB/s on intel core 2 Cycles Per Byte Adler32 920 1.9 MD5 255 6.8 SHA-1 153 11.4 SHA-256 111 15.8 SHA-512 99 17.7
Preserving data Practical usage of checksums Simple approach o compute checksum in memory when creating/writing file store checksum in a DB o check it in memory on full file reads Problems o corrupted data only found when read back,unnoticed otherwise o one needs to fully read the file to be able to check o file updates not supported.Need to read back the whole file o file append suffers the same limitation o multi stream,out of order writing not supported o losing the DB loses all checksums o renaming a file implies changing the entry in the DB,with double commit issue 17/ 3.ronce·ETrW
Preserving data 17 / 43 S. Ponce - CERN risks consistency safety c/c cksum block Practical usage of checksums Simple approach compute checksum in memory when creating/writing file store checksum in a DB check it in memory on full file reads Problems corrupted data only found when read back, unnoticed otherwise one needs to fully read the file to be able to check file updates not supported. Need to read back the whole file file append suffers the same limitation multi stream, out of order writing not supported losing the DB loses all checksums renaming a file implies changing the entry in the DB, with double commit issue