Holy Ten Point Eighty-Seven Jigga-Pixels, batman!

As I posted earlier, Randall has made a mental comic today. XKCD 1110. It is a tiled image that you click and drag to view. The problem is that it is huge!

Several people have played around with the tiles today, and done things somewhat more quickly than I did. I thought how hard can it be? to simply make the tiles in to one image.

The catch is that it is 165888 by 65536 (bigger if you include blank rows at top and bottom). That is 10871635968 pixels (10.87Gpix). (Do forgive the back-to-the-future title).

First thing I did was look at his javascript, work out the file names, and make a simple html page showing them all tiled. This is pretty browser crashingly big as it is, but did work. It only took  couple of minutes to make. The tiles have north/source and east/west components (no zeros), e.g. 1n1w.png, so easy to make a matrix. Some tiles are missing as they are all white (north) or all black (south).

So, first off, I just saved the html page, so it would save all of the images in a directory for me. Firefox had interesting ideas on how long this would take which are reminiscent of XKCD 612 but actually took quite a few minutes.

Once I had all of the images, I could easily add missing black and white images. Now to tile them - should be simple enough.

First I tried ImageMagick - one command telling it to tile all 2592 files. First I tried making smaller versions (one eighth each way). It eventually finished and went horribly wrong. Hmmm.

So them I tried GraphicsMagick. It would not do all files in one go. Just said "Killed" after a while. I suspect too many files.

So I tried making each row from the cells, and then making the rows to one file. The rows could be made in memory, just, and took around 20 seconds per row. Interestingly, the same thing on ImageMagick took 4 minutes per row. A win for GraphicsMagick, I think.

Then I told it to join the rows. It thought about this a lot, clocked up 20GB of virtual memory, and made a lot of temporary files. After a long time, it then freed up the virtual memory and made a 40GB temp file.

This is silly, it is 10Gpix of greyscale. I even did a -type greyscale to tell it that. Yet still, it is clear from the size and content of the temp file that it has made a 10Gpix file of 8 bit RGBA data (4 bytes per pixel). No wonder it is slow. How stupid! This does rather explain why it was running files on disk not just using RAM or swap.

Right now it is plodding through that, presumably packing it for the final png output. I'll post that once I have it, as a 10Gpix png greyscale image. Even so, this has taken all day. Don't worry, I was able to do some other work while this was happening, honest :-)

What will be interesting is to re-visit this in, say, 2 years. Will handling a 10Gpix file seem run of the mill, and just fit in RAM with no issues on a typical desktop machine. Probably! If this machine was not a year old it would probably be a lot slicker even now.

Update: The run to join the rows took 3 hours 20 minutes.

801.802u 631.023s 3:20:21.13 11.9%    0+0k 349656312+411601712io 181862pf+0w

The final file is 16MB (yes, that is all, lots of blank space) and is here.
Smaller version here and here.