The Lair

I’ve made several references too many about the large amount of data that I need to crunch for my work. Well, here’s another … It’s contained in text files that are first bzipped and then tarred. Mere details, you might think… except that the tar file always refuses to decompress properly.

Trivial, I hear you say… just download the darn thing again and try decompressing it. This is gigabytes of data though, so ummm.. downloading it again takes about half an hour. No biggie. Download, try decompressing and you have the same result. My trusty WinRAR says the archive is corrupt.

So I have lots of data, in a corrupt archive and I can’t salvage anything from it.

Try the freeware 7-Zip archiver instead and it chokes and dies at the size of the file.

Back to square one. So, command line tar to the rescue. I tell it to do its thing on the gigantic archive and it actually runs through the entire file without complaining. Maybe tar just silently skips the parts that it can’t extract, I have no idea.

Was it a bug in WinRAR ? I don’t know and I’d rather not find out. But I have some test data. Wooo. So much for rigourous attention to detail. I’m treating this test data corruption episode as “what I don’t know won’t hurt me… much”.

And in keeping with the principle of uncertainty, an old bug report from Microsoft Visual C++ that makes interesting reading (via kashmera)