[TSTIL] Instadeploy

First published on

[ This is a part of "The Software That I Love", a series of posts about Software that I created or had a small part in ]

2015 - InstaDeploy

When we launched "Free Apps" at Mendix I was pretty proud. It was basically "Sandboxes V2 + WebModeler". Under the hood we had this whole new architecture with all kinds of clever "cloud-scale" solutions. It turned out that users were not so happy. Every time you changed something in your app you had to deploy it. We wanted people to use Free Apps so we hid the old "local deployment" option. The Free Apps deployment service was buggy and slow. Local deployment used to take 10 seconds. The old sandboxes took 1 minute, the Free Apps took 4. On bad days it took 8.

The cloud team had launched the deployment service without any SLAs or basic explanation of the limitations to the other teams. A bit immature. The WebModeler and Modeler teams just had to use it and see how it worked. In testing it was already very flaky, but under production usage it was really bad. We thought we could get away with it because in the cloud, the rubber hits the road and we had failing stuff all the time. The other teams had stable unit tests etc. Not us. We thought that was cool. Now the ones who got most of the flak from management was the WebModeler team. They had all the visibility and the deploy button was in their product. They felt helpless.

Me and the cloud guys were not very open to helping them out. I was under a lot of stress to launch cloud v4. In the meantime we were scaling cloud v3 and needed new hardware every couple of months. This took most of my time and mental energy. I also could not justify the focus on our Free Tier while our paid production workloads were in danger. Sorry for having to deal with that, Erik vd P., Michel, Kishen, Daniel D. and Arjan B. . After it got really bad we took part of the blame and we started figuring out how to optimize the deployment time.

Then we had the annual Company Kick-Off in Noordwijk (the Mendix CKO). I ran into Derek, and said that I was confident we could get it to two minutes. His response was very Steve Jobs-like: "Two minutes? Why should it take any time at all? It should feel instant! If I make change I want to see it right away!". F*ck. He was right. We needed to rethink our entire approach.

I've never had a clearer call to action. I don't know how, but I managed to convince Johan that we had to put the top hackers from every team in one room for two days. This was Benny for the Modeler/MxBuild, Michel for the WebModeler, Xiwen and Daniel for Cloud, and maybe some more people. Let me know if you were there. I was there too, "managing". Because of Derek's speech I was fired up. Everyone else was happy we could finally fix the deployment problems. We all felt like we were on a mission from God.

We added some timing logs, broke down the deployment process and started optimizing our own separate parts of it. We started at 4 minutes and had ideas on how to get to 2. I realized we had to take ugly shortcuts to get beyond that. Instead of sending the deployment package to the deployment service, it had to go straight to the running container. This was very unholy and the 12-factor-app gods did not approve. I thought of Bill O'Reilly. "Fuck it, we'll do it live". We modified the buildpack to include a compiler on hot standby. When it got a deployment package on a separate secret HTTP path, it would trigger the compiler and reload the app. Benny built the mxbuild server. Xiwen got it to run in the container. Michel optimized the webmodeler so it could create a deployment package in 10 seconds instead of 20.

The time kept dropping. We were at 40 seconds. Benny optimized some more with pre-warming the build cache. Michel said "a-ha" and got package creation down to 0.1 seconds. At night Xiwen or me finally got mxbuild to run. On the second day the end-to-end unholy deployment cycle was working. Not in 2 minutes, not in 1 minute, but in 2 seconds. Holy shit.

It made for a great demo, a great story and when it worked, a great product. It never got super reliable, but boy was it fast.






Comments

Post a Comment

Popular posts from this blog

"Security Is Our Top Priority" is BS

First published on
A couple of years ago I was asked to give a conference talk about software security. Well, actually I wasn't really asked, my company bought a sponsorship package with a speaker slot and I replied to the internal email asking for volunteers  🤣  Anyway, while preparing my talk, I realized a couple of important points about security that have not left my mind since: Security is limitless . You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc. The needs of security are opposed to the needs of a convenient user experience . Improving one typically hurts the other. Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that
Read more

AI programming tools should be added to the Joel Test

First published on
Here's a wake-up call to all CTOs: AI programming tools are getting freaking amazing and if you don't allow your teams to use them somehow, it will bite you in the ass in a couple of years. You will be slower and you will lose your best people. The infamous Joel Test is a list from the year 2000 of 12 things all great software companies do. Since then most companies have implemented Git and CI/CD, checking of three items, so we have some space left in the 2024 update ;) I believe "Do you allow your developers to use AI assisted development environments?" is a necessary addition. I get that you don't want your source code to end up on some OpenAI / Microsoft / Github server somewhere, sure, but find a way to use your own models or learn to live with it. Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money. So why do I think developers need AI programming tools? I&#
Read more

The unreasonable effectiveness of i3, or: ten years of a boring desktop environment

First published on
Image
My wife uses Windows and over the years I've helped her move things to new systems. Win8, 10 and now 11. With every upgrade the UI changes. Now I can't right click the bottom right corner anymore to open Task Manager. The UI feels "fresh" and up-to-date I guess, but does it really matter? My desktop has looked like this since 2008. I love the picture of the gearbox, it's a testament to the hidden precision engineering that goes on inside the built world all the time. I don't see much of the gearbox though, because most of the time my screens look like this: The environment around my background picture has changed a little more, but has been stable for the last 10 years. My experience is very different to my wife's constantly changing system. In 2009 I moved from Windows to Ubuntu, then to Debian using Gnome. Then finally to i3 in 2013 after a brief affair with XMonad in 2012. So by now in early 2024, I've had the same minimal UI for more than 10 years,
Read more