ConfigureHow

Bit rot, checksums, and the copy nobody checked

By Timor Nagy. Published . 8 min read

Silent data corruption is real, it has been measured across more than a million drives, and per drive per year it is roughly twenty times rarer than the read errors your disk already reports out loud. The way a file usually breaks without anyone noticing is duller than decay: a copy or a sync ran, something went wrong partway through, and nobody compared the result against the original. A checksum is the cheap fix for that second problem and the only detector for the first, and most people who worry about bit rot have never once verified a transfer.

What silent corruption is, precisely

The reference definition comes from Bairavasundaram, Goodson, Schroeder, Arpaci-Dusseau and Arpaci-Dusseau at USENIX FAST '08. Ordinary bad sectors are not silent: "Latent sector errors are detected by a drive's internal error-correcting codes (ECC) and are reported to the storage system." Silent corruption is the case where data is "silently corrupted with no indication from the drive that an error has occurred."

Their Storage Developer Conference 2008 talk on the same data splits it into four shapes: bit corruption, lost writes where "data not written but completion is reported", misdirected writes landing on the wrong block, and torn writes partially written and reported complete. The slide's closing line matters: "In all cases, data passes disk's internal ECC."

Three of those four are not media decay at all. They are firmware or software claiming a write happened that did not. Magnetism quietly flipping on a platter is the story people tell; a controller lying about a write is what the field data mostly shows.

The two studies everyone cites, and what they leave out

The FAST '08 paper analysed 1.53 million drives over 41 months from January 2004: "We find more than 400,000 instances of checksum mismatches over the 41-month period." That sentence travels the internet on its own. The denominators do not.

Of those 1.53 million disks, 3,855 developed any mismatch at all, 3,088 of the 358,000 nearline SATA disks (0.86%) and 767 of the 1.17 million enterprise Fibre Channel disks (0.065%). "On average, each disk developed 0.26 checksum mismatches." The distribution is brutally uneven: "1% of the corrupt disks (the top 1% corrupt disks with the largest number of mismatches) produce more than half of all mismatches recorded in the data", and "The maximum number of mismatches observed for any single drive is 33,000." A handful of sick drives, not a slow universal rot.

The CERN measurement gets the same treatment. Bernd Panzer-Steindel's Data integrity note, draft 1.3 dated 8 April 2007, checked stored files against previously computed adler32 values: "During a test 33700 files were checked (~8.7 TB) and 22 mismatches found. That translates into an error rate of one bad file in 1500 files." That ratio is quoted constantly. The next paragraphs of the same document are not. A separate probe writing and re-reading 2 GB patterns found that 80% of its errors were "64k regions of corrupted data" with "large correlation with the 3ware-WD disk drop-out problem", which "was identified as a problem in the WD disk firmware", and CERN was "currently updating the firmware of about 3000 disks". Single bit errors were 10%. One in 1500 was a snapshot of a specific broken controller and drive pairing in 2007, not a decay constant for your photo library.

What the drive already does for you

Every sector is stored with error-correcting code, and the drive reports what it cannot fix. Seagate's IronWolf Pro data sheet quantifies that as "Nonrecoverable Read Errors per Bits Read, Max" of "1 per 10E15". Note the word nonrecoverable. Those errors are loud: the read fails and you get an I/O error, which is exactly why they are not the thing to be afraid of.

The FAST '08 comparison table puts both classes on the same scale, as the average percentage of disks affected per year. Latent sector errors hit 9.5% of nearline and 1.4% of enterprise disks. Checksum mismatches hit 0.466% and 0.042%.

Reported disk errors against silent ones, per drive per year A logarithmic bar chart of the average percentage of disks affected per year in the FAST 2008 field study of 1.53 million drives. Latent sector errors, which the drive detects and reports, affected 9.5 percent of nearline SATA disks and 1.4 percent of enterprise Fibre Channel disks. Checksum mismatches, which the drive does not report at all, affected 0.466 percent and 0.042 percent of the same two populations. The reported errors are about twenty times more common than the silent ones on both disk classes. Disks affected per year: what the drive tells you, and what it does not Averages from the same 1.53 million drive population. The scale is logarithmic, one decade per gridline. 0.01% 0.1% 1% 10% Reported by the drive: latent sector errors Nearline 9.5% Enterprise 1.4% Silent: checksum mismatches Nearline 0.466% Enterprise 0.042% Nearline means SATA, enterprise means Fibre Channel, in the vocabulary of the 2008 paper. A bar left of the 0.1% line is an error you will never meet on one drive in one year.
Per drive per year, the errors a disk reports are more than twenty times as common as the ones it hides.

Why a checksum is the only proof that a copy worked

Copy tools compare far less than most people assume. The rsync manual is explicit about its default: "rsync uses a 'quick check' that (by default) checks if each file's size and time of last modification match between the sender and receiver." Matching size and timestamp means the file is skipped and its contents are never read on either side. Rsync does verify what it decides to send, and says so: "rsync normally verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred."

On Windows the trap is a letter. Robocopy's /v is a logging switch, documented as "Produces verbose output, and shows all skipped files." The older xcopy is the one whose /v actually means verify: "Verifies each file as it is written to the destination file to make sure that the destination files are identical to the source files."

How deep each post-copy check actually reads A copy runs from a source file through a cable, network and controller to a destination file, and can fail in the middle through a torn write, a lost write, a dropped block or a full volume. Two checks are then compared. The default quick check compares only size and modification time, reads no file contents at all, and skips a damaged copy that happens to match on both. A whole file checksum reads every byte on both sides, so any single changed byte fails, at the cost of one full read of each tree. A copy is only verified if something read both sides The default check in most sync tools never opens the file. Source file Cable, network, controller Destination file torn write, lost write, dropped block, full volume what the check reads what it can tell you Quick check rsync default Size and time only Skipped, bytes never read same size and time passes untested Whole file checksum rsync -c, sha256sum -c Every byte, both sides Any changed byte fails costs one full read of each tree Metadata only Contents read and compared On Windows the verifying switch is xcopy /v. Robocopy /v is verbose output, not verification.
The default comparison never reads the bytes, which is why a damaged copy passes it.

Fast hash or cryptographic hash

Against accidental damage, any decent hash works, and a 64 bit digest leaves about one chance in 1.8 x 10^19 that a random change produces the same value. Speed is then the only thing that matters, and the gap is enormous. On the xxHash project's own benchmark, an Intel i7-9700K running clang 10 with -O3:

AlgorithmDigestBandwidthProject's own note
XXH3 (SSE2)64 bit31.5 GB/sfaster than the 28.0 GB/s reference RAM read
XXH128 (SSE2)128 bit29.6 GB/s
XXH6464 bit19.4 GB/s
BLAKE2256 bit1.1 GB/s"Cryptographic"
SHA1160 bit0.8 GB/s"Cryptographic but broken"
MD5128 bit0.6 GB/s"Cryptographic but broken"

The cryptographic column buys one extra property: resistance to somebody deliberately constructing a different file with your hash. MD5 and SHA-1 no longer have it. NIST announced on 15 December 2022 that SHA-1 "should be phased out by Dec. 31, 2030", because "Today's more powerful computers can create fraudulent messages that result in the same hash as the original". Microsoft draws the same line in the Get-FileHash documentation: MD5 and SHA1 "should only be used for simple change validation, and should not be used to generate hash values for files that require protection from attack or tampering".

So: verifying your own copy of your own drive, use xxHash and finish in a fraction of the time. Verifying a download against a hash a stranger published, use SHA-256 or BLAKE3, which its authors describe as "Secure, unlike MD5 and SHA-1. And secure against length extension, unlike SHA-2."

Verifying a transfer on Windows, macOS and Linux

The pattern is a manifest written at the source and checked at the destination.

Linux, with coreutils: find . -type f -print0 | xargs -0 sha256sum > SHA256SUMS on the source, then sha256sum -c SHA256SUMS at the destination. The -c flag will "Read file names and checksum information (not data) from each file ... and report whether the checksums match the contents of the named files." Add --quiet, which suppresses the per-file OK line so only failures print.

macOS has shasum, whose -a takes "1 (default), 224, 256, 384, 512, 512224, 512256". The default is SHA-1, so always pass -a 256: shasum -a 256 file to generate, shasum -a 256 -c SHA256SUMS to "read SHA sums from the FILEs and check them".

Windows: Get-FileHash -Path file already defaults to SHA256. For a one-off from cmd, certutil's -hashfile "Generates and displays a cryptographic hash over a file", invoked as certutil -hashfile InFile [HashAlgorithm].

For a whole tree, a second rsync pass with -c is less work than a manifest. It "changes this to compare a checksum for each file that has a matching size", which reads both trees end to end. Pin the algorithm with --checksum-choice, which accepts xxh128, xxh3, xxh64, md5, md4, sha1 and none. Never none: the manual warns that then "the --whole-file option is forced on and no checksum verification is performed on the transferred data."

What ZFS and Btrfs buy, and what they do not

Both compute a checksum for every block and validate it on every read, which converts silent corruption into a loud error. Neither can repair anything on its own.

ZFSBtrfs
Default algorithmfletcher4crc32c
Others availablesha256, sha512, skein, edonr, blake3xxhash (64 bit), sha256, blake2b, since kernel 5.5
Checksum storedin the block pointer, away from the datain a separate checksum tree
Repair requiresRAID protection or the copies propertya replicated profile such as RAID1

OpenZFS states the condition plainly: corrupt blocks "are automatically repaired if possible, by using the RAID protection in suitably configured pools, or redundant copies". The Btrfs manual is blunter about the limit: scrub "can only repair filesystem damage by copying from other known good replicas". On a single disk with a single profile you get detection and a clear error, which is a real gain over a file that opens and looks wrong, but it is not recovery.

Which layer detects corruption and which one can undo it Four layers, each paired with what it can actually do. The drive's own error correcting code corrects what it can and reports the rest, but silent corruption passes it because the drive sees nothing wrong. A checksumming file system such as ZFS or Btrfs validates every block it reads and returns an error instead of bad data, which is detection without repair. Redundancy, meaning a mirror, raidz, a Btrfs RAID1 profile or extra ZFS copies, is the first layer that can rewrite the damaged block from a verified replica. A backup restores what all three above have already lost, but only if the restore has been tested. Detection and repair are different layers mechanism what it can do Drive ECC in every sector Corrects, then reports Silent corruption passes the drive sees nothing wrong Checksum ZFS, Btrfs Validates every block read Error instead of bad data detects, cannot repair alone Redundancy mirror, raidz, RAID1 Copies a verified replica Repairs the damaged block needs a good copy to copy from Backup read-data verified A second, separate copy Restores what is lost only if the restore was tested Reports only Detects Repairs Restores A single disk with no replica stops at the second row: you learn the block is bad, and that is all.
Detection and repair are separate jobs, and only the redundancy layer can do the second one.

Scrubbing, and how often

A scrub reads everything and checks it while you still have redundancy to fix it with. zpool scrub "examines all data in the specified pools and verifies each block's checksum", and "For replicated (mirror, raidz, or draid) devices, ZFS automatically repairs any damage discovered during the scrub." Btrfs describes scrub as "a validation pass over all filesystem data and metadata" and gives a schedule: "The recommended period is a month but it could be less."

That cadence is not arbitrary. In the FAST '08 data, scrubbing was the single largest detector: "on the average data scrubbing discovers about 49% of checksum mismatches in nearline disks", and 73% in enterprise ones, on systems where "an entire RAID group is scrubbed approximately once every two weeks on an average". The mismatches a scrub does not find surface during RAID reconstruction instead, when the redundancy that would have repaired them is already spent.

An unverified backup is not yet a backup

Backup programs default to cheap consistency checks that never read your data. Restic says so: "By default, the check command does not verify that the actual pack files on disk in the repository are unmodified, because doing so requires reading a copy of every pack file in the repository." Only restic check --read-data reads it. Borg is the same shape: --verify-data "will perform a full integrity verification (as opposed to checking the CRC32 of the segment) of data, which means reading the data from the repository, decrypting and decompressing it."

Reading an entire off-site repository every month is expensive in time and, on metered storage, in egress. Restic's --read-data-subset takes an n/t form that splits the repository into t groups and reads only group n. Run 1/12 this month through 12/12 next December and every byte has been read once a year at a twelfth of the cost per run. Put that on a schedule, because the failure mode of every technique above is never that it was wrong, only that nobody ran it.

Sources