[TSTIL] JUDO

[ This is a part of "The Software That I Love", a series of posts about Software that I created or had a small part in ]

2012 - JUDO

At Mendix I worked on the Cloud team, we hosted Mendix apps for customers. Hosting software should be as easy as building software. The Cloud/Ops/DevOps bunch was a tightly-knit team, as we were in the firing line together. With that I mean 24/7/365 uptime guarantees and you need to rely completely on your colleagues at all times of the day. It's something else. When I joined we were 4, when I left it was 12. Now it's 100. I'm in touch with a lot of these people to this day.

Every night the system created backups of the apps which admins could download over HTTP. The download consisted of the postgres dump, but also a lot of files from object storage. We could store ZIPs for every backup every night, but then the storage requirements would explode. Instead, we did incremental backups with rsync and hardlinks. Achiel wrote his BSc. thesis on this project for the sole reason of graduating within 10 years of starting college. If not, he would have to pay back his government study subsidies. That was fun. Hans was actually the mastermind behind the backup system. He's an incredible engineer and always built rock-solid stuff. Now this system was pretty cool, but the download needed to be a single zip file and that was difficult.

We built JUDO, the Java Upload Download Organizer. When getting a download request over HTTP, it assembled the zip file on-the-fly while walking over the files storage directories. This was a pretty complicated project and Xiwen, Frank and I needed a lot of help from the senior engineers (Hans and Achiel). Over the years we found obscure bugs and fixed most.

The streaming approach was very elegant. There was no temporary storage and no asynchronous processes. The fundamental problem was that the size of the file was not known at the start. The size and Content-Length header could not be predicted because of the compression. When a connection got interrupted, browsers were not sure whether the download was complete. This led to some problems. This is why most products use a "we are preparing your download" approach. Less efficient, but when dealing with backups it's important to know you have downloaded it completely.





next2012 - Decrypt.py

previous2012 - autopullpep8 and the facebook adventure



Comments

Popular posts from this blog

AI programming tools should be added to the Joel Test

The unreasonable effectiveness of i3, or: ten years of a boring desktop environment

The long long tail of AI applications