Wow, a topic by me. I don't usually do that.

Using LZMA compression, I've managed to reduce the file size of various maps. This isn't the same as putting a map in a .7z file, as the header of the map still remains uncompressed. Rather, it's more comparable to compressed Xbox maps. That said, it does take a little longer to load a compressed map than an uncompressed map. Having more CPU cores and threads can potentially mean better decompression times, but this will vary from map to map, as will compression ratio.

For example, the entire Refined campaign takes up 2.649 GiB normally, but as compressed maps, it takes up only 1.003 GiB. A more extreme example is With Whiteness which, when uncompressed, is 759.7 MiB, but compressed results in a file size of 65 MiB. For decompression time, the largest map in the Refined campaign, d40.map, takes around 1.67 seconds to decompress when allowed a thread count of 8 or more, but with fewer threads, the time goes up significantly. I'll include this map and several other maps in a table. As for With_Whiteness, because most of the data is just unused 00's, the decompression time is very fast: 0.89 seconds at 12 threads, 1.26 seconds at 8 threads, 1.71 seconds at 4 threads, and 3.12 seconds with 2 threads.

Not everything works with this, however. Open Sauce maps, most protected maps, and maps with embedded Chimera scripts cannot be used with this, and most of this is, unfortunately, technically beyond my control.

Here is a table with some benchmarks:



The benefit to this is that large map folders can use less space without having to be stored in an archive like a .rar or .zip. Also, you do not have to cut back on image quality nearly as much. The loading time penalty is minor, and it can be made even less apparent by caching recently used maps.

Edited by Kavawuvi on Mar 11, 2019 at 12:28 AM

that's cool, how does it work? does a header flag tell the game it's compressed and it just reads the raw compressed data after the header? or is it done in blocks?

Actually, either work! The header contains a few fields: the block count and compression algorithm. Currently only uncompressed and LZMA are supported.

If the block count is 0, then all of the data after the block is considered one stream of data. This is like what Xbox does. This is what I use for maps that are under 8 MiB when uncompressed. Obviously uncompressed maps don't use blocks, too, so it will be set to 0 here.

If the block count is nonzero, then there is a table located directly after the header containing information about each block (decompressed size, compressed size, and file offset) with the number of entries equal to the block count. In decompression, this data is then decompressed and reconstructed into the original map file.

The benefit to doing blocks is that it can be easily split up into threads, and this massively reduces compression and decompression time while only slightly affecting the compression ratio.

Whether or not to use blocks as well as the block size to use is determined by the tool based on the file size, but it can be overridden. Larger maps use larger blocks, smaller maps use smaller blocks.

Edited by Kavawuvi on Mar 10, 2019 at 04:18 PM