[TSTIL] The Reaper

First published on

[ This is a part of "The Software That I Love", a series of posts about Software that I created or had a small part in ]

2017 - The Reaper

We launched Mendix Cloud v4 by starting the new Free Apps tier on it. These apps without SLA allowed us to take more risks and learn to operate Cloud Foundry. The Mendix Free Apps cluster had enough RAM for about 100 concurrent apps. It could scale up and down, but we wanted to keep the costs reasonable.

An app would run as long as there was HTTP traffic to it. If an app did not have any traffic for 30 minutes, we killed it. If an HTTP request would come in, we'd serve a "loading" page while the app was spinning back up.

We had a small app (The Resumer) that would catch all traffic for apps that were not running & serve the loading page. I believe Xiwen wrote most of it, but my memory is failing me. It would fire a request to the CF API to find and start the app.  The Reaper was its counterpart. It would talk to the CF API, list all running apps. Then call the admin API of the app to see when its last request was handled, and kill it if > 1800 sec. For some reason, The Reaper stopped running once we had more than 15.000 (?) apps in the Free Tier. The cluster started misbehaving badly when out of capacity. We could not figure out why or how, but needed a fix.

When running the same code on my workstation everything worked fine and the cluster went back to normal. So, you can guess what I did. Against all proper engineering practices, I ran "The Reaper" from the workstation under my desk with a cron job for a couple of months. I was a Product Manager and should not be touching code or infrastructure at all, but I couldn't help myself. Later on, Daniel vD (now also a CTO), found that there was a bug in the CF API logic. Without admin credentials it would make a query that scaled O(n2) with the number of apps. Admin credentials didn't need the extra security protection for the query so that's why that was fast. As we were the only idiots in the world running Cloud Foundry with 15k stopped and 100 running apps, no one else had run into this.

Thanks to my friend Daniel you can read the issue here: https://github.com/cloudfoundry/cloud_controller_ng/issues/1272  . When he documented the issue we were apparently at 85k apps. We were growing fast! The Reaper was a pretty cool name for a pretty cool tool.







Comments

Post a Comment

Popular posts from this blog

"Security Is Our Top Priority" is BS

First published on
A couple of years ago I was asked to give a conference talk about software security. Well, actually I wasn't really asked, my company bought a sponsorship package with a speaker slot and I replied to the internal email asking for volunteers  🤣  Anyway, while preparing my talk, I realized a couple of important points about security that have not left my mind since: Security is limitless . You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc. The needs of security are opposed to the needs of a convenient user experience . Improving one typically hurts the other. Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that
Read more

AI programming tools should be added to the Joel Test

First published on
Here's a wake-up call to all CTOs: AI programming tools are getting freaking amazing and if you don't allow your teams to use them somehow, it will bite you in the ass in a couple of years. You will be slower and you will lose your best people. The infamous Joel Test is a list from the year 2000 of 12 things all great software companies do. Since then most companies have implemented Git and CI/CD, checking of three items, so we have some space left in the 2024 update ;) I believe "Do you allow your developers to use AI assisted development environments?" is a necessary addition. I get that you don't want your source code to end up on some OpenAI / Microsoft / Github server somewhere, sure, but find a way to use your own models or learn to live with it. Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money. So why do I think developers need AI programming tools? I&#
Read more

The unreasonable effectiveness of i3, or: ten years of a boring desktop environment

First published on
Image
My wife uses Windows and over the years I've helped her move things to new systems. Win8, 10 and now 11. With every upgrade the UI changes. Now I can't right click the bottom right corner anymore to open Task Manager. The UI feels "fresh" and up-to-date I guess, but does it really matter? My desktop has looked like this since 2008. I love the picture of the gearbox, it's a testament to the hidden precision engineering that goes on inside the built world all the time. I don't see much of the gearbox though, because most of the time my screens look like this: The environment around my background picture has changed a little more, but has been stable for the last 10 years. My experience is very different to my wife's constantly changing system. In 2009 I moved from Windows to Ubuntu, then to Debian using Gnome. Then finally to i3 in 2013 after a brief affair with XMonad in 2012. So by now in early 2024, I've had the same minimal UI for more than 10 years,
Read more