Archive formats and compression: what actually decides the choice

The compression ratio a format advertises is the least useful number for choosing one. Ratio decides things when you are paying for petabytes or pushing bytes down a metered link. The archive you are about to hand to a colleague is decided by whether their machine opens it, whether it carries the permissions and timestamps you need at the far end, and how much of it you lose when one byte goes bad. Every format answers those three clearly, and none of the answers appear in a benchmark table.

Archiving and compressing are two jobs, not one

The gzip manual states the division of labour outright. For "a single archive file with multiple members so that members can later be extracted independently" it sends you to "an archiver such as tar or zip", and concludes: "gzip is designed as a complement to tar, not as a replacement."

RFC 1952, version 4.3, May 1996, defines gzip as "a compression method and a file format (the latter assuming only that a file can store a sequence of arbitrary bytes)". No member list, no directory tree, no permission bits. Its trailer holds a CRC32 of the uncompressed data and an ISIZE field carrying "the size of the original (uncompressed) input data modulo 2^32". That check covers the stream, not any file inside it.

tar supplies the structure gzip refuses to. Its current definition is the pax utility in the Open Group Base Specifications Issue 8, IEEE Std 1003.1-2024. ustar shows its age in its field widths: 100 octets for the name, 155 for the prefix, so "pathnames of at most 256 characters can be supported", with a size field topping out at 8589934591 octets. The pax interchange format escapes those limits with extended header records, and its timestamps carry fractional seconds, where "the digits to the right of the point shall represent the units of a subsecond timing granularity".

The consequence people meet by accident: a .tar.gz has no index, so listing it means decompressing all of it.

What a zip file actually is

Zip inverts the arrangement, and PKWARE's APPNOTE.TXT version 6.3.10, dated 1 November 2022, states the model in one sentence: "Each data file placed into a ZIP file MAY be compressed, stored, encrypted or digitally signed independent of how other data files in the same ZIP file are archived." Every entry gets its own local header and its own compressed stream, and a central directory at the end of the file indexes them.

That independence is why a zip viewer lists an archive instantly and pulls one file out of ten thousand without touching the rest. It is also why zip loses on ratio to a solid archive: the compressor's history resets at every file boundary, so structure repeated across similar small files is invisible to it.

Zip is a container, not an algorithm. Method 8 is deflate, what almost everything writes. The same spec assigns method 12 to bzip2, 14 to LZMA and 93 to Zstandard. A zip built with method 93 is a valid zip file that most tools refuse to open, which is the format's recurring trap: universal container, non-universal contents.

Three numbers, and only one of them is ratio

The zstd project publishes its benchmark with the methodology attached: "several fast compression algorithms were tested and compared on a desktop featuring a Core i7-9700K CPU @ 4.9GHz and running Ubuntu 24.04 (Linux 6.8.0-53-generic), using lzbench, an open-source in-memory benchmark by @inikep compiled with gcc 14.2.0, on the Silesia compression corpus." Silesia is 211,938,580 bytes assembled to "provide a data set of files that covers the typical data types used nowadays".

Read down the ratio column and the three look close. Read across and they do not. lz4 gives up about 27% of zstd's ratio and returns 2.5 times the decompression throughput. zlib at level 1 is slower than zstd at level 1 and produces a bigger file, which is what ended deflate's default status in a lot of places.

You compress once and decompress many times, and the zstd manual states the relevant property directly: "It also features a very fast decoder, with speeds > 500 MB/s per core, which remains roughly stable at all compression settings." Raising a zstd level buys ratio with compression time and costs nothing at extraction. LZMA does not behave that way.

Why zstd took the distribution and backup slots

Arch Linux switched package compression on 2020-01-04 and published the arithmetic: "Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup."

Fedora measured a Firefox installation both ways for the Fedora 31 change. xz level 2 produced 1,615,017,616 bytes; zstd level 19 produced 1,575,843,696 bytes, so the higher-ceiling format lost on size too. Install time on tmpfs went from 8s to 2s, and on ext4 over nvme from 11s to 4s.

zstd's default is level 3 on a documented scale of "[1-19]", with a separate flag needed above 19 and a ceiling of 22. Negative levels run the other way: "The negative compression levels, specified with --fast=#, offer faster compression and decompression speed at the cost of compression ratio."

xz, and the memory bill nobody reads

xz buys ratio with memory, asymmetrically: "Typically the decompressor needs 5 % to 20 % of the amount of memory that the compressor needed when creating the file." The default is deliberately conservative: "-6 is the default, which is usually a good choice for distributing files that need to be decompressible even on systems with only 16 MiB RAM." The top presets do nothing for small inputs, because a dictionary larger than the file cannot help: "These are useful only when compressing files bigger than 8 MiB, 16 MiB, and 32 MiB, respectively."

The .xz container is better engineered than gzip's. Version 1.2.1 of the format specification, dated 2024-04-08, defines four check types, "None, CRC32, CRC64, and SHA-256", with CRC64 the default. It also stores an index, so a decoder can "quickly access the beginning of any Block (random access)".

xz earns its place where a file is compressed once on a build machine and downloaded many times.

7z

7-Zip describes its own format briefly: "7z is the new archive format, providing high compression ratio", with five methods integrated, "LZMA, LZMA2, PPMD, BZip2, Copy". For the zip and gzip files it writes it claims a "compression ratio that is 2-10 % better than the ratio provided by PKZip and WinZip". Version 26.02 shipped 2026-06-25. The 7z format's advantage over zip is solid compression: files go into one continuous stream, so cross-file redundancy reaches the matcher.

Corruption blast radius

GNU tar's manual is blunt about what tar checks: "An tar-format archive contains a checksum that most likely will detect errors in the metadata, but it will not detect errors in the data." It catches a mangled header, not a flipped bit inside a file.

Wrapping tar in gzip adds a CRC over the whole stream, which tells you the archive is broken without telling you where, and damage early in a solid stream costs everything after it. gzip's only containment mechanism is concatenation: "If one member is damaged, other members might still be recovered after removal of the damaged member." A normal .tar.gz is a single member.

bzip2 is the outlier, which is why it ships a recovery tool. Its manual: "Each block is handled independently" and "Each block also carries its own 32-bit CRC, so damaged blocks can be distinguished from undamaged ones". bzip2recover exists to "search for blocks in .bz2 files, and write each block out into its own .bz2 file". Block size is selectable from 100 k to 900 k.

The metadata row is the one that surprises people. Zip stores times in "standard MS-DOS format", and MS-DOS "uses year values relative to 1980 and 2 second precision": no timezone, no sub-second field. Permissions live in external file attributes whose "mapping of the external attributes is host-system dependent". Extra fields 0x5455 and 0x7875 patch the holes, when both ends implement them.

Encrypting the contents is not encrypting the file list

Zip's original encryption should not be used. Info-ZIP's own manual page says to "use strong encryption such as Pretty Good Privacy instead of the relatively weak standard encryption provided by zipfile utilities". The replacement is WinZip's AES scheme: "The encryption specification supports only 128-, 192-, and 256-bit encryption keys. No other key lengths are permitted." Its AE-2 variant removes a leak: "For files encrypted using the AE-2 method, the standard Zip CRC value is not used, and a 0 must be stored in this field."

None of it hides the central directory, so every file name, every size and the whole directory tree stay readable to anyone holding the archive. APPNOTE defines a Central Directory Encryption feature for precisely this, and when its masking bit is set "the file name stored in the Local Header will not be the actual file name", but support outside PKWARE's own products is close to nonexistent. 7z solves it with one switch, documented in the 7z manual page as writing an archive "with data and header archive encryption on".

An encrypted archive with a deliberately bland filename whose entry list reads settlement-draft-v9.docx has leaked the part that mattered.

Deduplicating backup tools moved the goalposts

For anything you archive repeatedly, ratio per archive is the wrong unit. What you pay is bytes added per run. BorgBackup: "Deduplication based on content-defined chunking is used to reduce the number of bytes stored: each file is split into a number of variable length chunks and only chunks that have never been seen before are added to the repository." It also "does NOT depend on: file/directory names staying the same". Compression is a per-chunk choice of lz4, zstd, zlib or lzma, and "All data can be protected using 256-bit AES encryption".

restic documents its cut points: "The data from each file is split into variable length Blobs cut at offsets defined by a sliding window of 64 bytes. The implementation uses Rabin Fingerprints for implementing this Content Defined Chunking (CDC)." "Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB in size." The payoff is a property no archiver has: "This even works if bytes are inserted or removed at arbitrary positions within the file." Everything stored is "encrypted with AES-256 in counter mode and authenticated using Poly1305-AES".

Insert one byte at the front of a 4 GB file and every fixed block offset after it shifts. A fresh .tar.gz of that directory is 4 GB of new archive at any compression level. A restic or Borg run re-uploads only the chunks whose content changed.

The recipient's toolbox is a hard constraint

Windows caught up recently, and only partly. A Windows 11 configuration update dated 26 September 2023 "adds native support for reading additional archive file formats using the libarchive open-source project", covering .tar, .tar.gz, .tar.bz2, .tar.zst, .tar.xz, .tgz, .tbz2, .tzst, .txz, .rar and .7z. The same note adds the limit that undoes most of that for anything sensitive: "This features does not support password encrypted files."

So a deflate zip opens everywhere with nothing installed, tar.gz and tar.zst open on any current desktop, and an encrypted 7z still requires the recipient to install 7-Zip. Shipping a level 19 archive nobody at the far end can open is a worse outcome than shipping deflate.