The software that I love

[ this is a long article, almost the size of a short book ]

Pride. When interviewing candidates I always ask "tell me something that you are proud of." Sometimes you get a blank stare. That person is down and dealing with some shit. Luckily most of the time the eyes light up and the person starts talking about them at their best. I love those moments. As in Peter Thiel's "From 0 to 1", I want to see the best in people and then see if we can deal with the bad. In the end, I ask "so this is all great, but what will we have to learn to live with if we hire you?". On the one hand I'd like to prepare myself, but the question tests for self-reflection too. It works well.

But back to proud. I've been dealing with some shit too, and there have been a bit more downs than ups lately. This blog post/diary is a way to make my eyes sparkle and remind myself of the things I have built and still love. These are short stories in chronological order. There's a lot more to life than software, but this focused format is simple and it works. I might write from other perspectives in other posts. I've written this post for myself, but if you enjoy it, that's all the better.

These stories are how I remember them. My memory isn't amazing and in most of them I only had a small part with a limited perspective. If you have additions, questions or other feedback, send me a message at jouke@waleson.com.

Note 1: This is a work in progress and will be updated when I have time.

Note 2: Wow, this post got long but also popular. It had over 10k views on day 1. To make it easier here are some links:

  • Part 1 - Fun but somewhat forced programming ('96-'00)
    • [added Oct 2.] 1996 - JTJ Rekenen
    • [added Oct 2.] 1997 - Plotter
    • [added Oct 2.] 1999 - Reverse engineering the binary format of HP ChemStation
  • Part 2 - High School & Uni ('00-'11)
    • [added Oct 2.] 2004 - The HP 49g
    • [added Oct 3.] 2005 - Porting an Atari tax program to Delphi
    • [added Oct 2.] 2009 - SnelTrein
    • [added Oct 30.] 2010 - Snapp
  • Part 3 - The Mendix Years ('11-'18)
    • [added Oct 8.] 2011 - newnode.py
    • [added Oct 2.] 2011 - Social Gravity
    • [added Oct 2.] 2012 - autopullpep8 and the facebook adventure
    • [added Oct 2.] 2012 - JUDO
    • [added Oct 30.] 2012 - Decrypt.py
    • [added Oct 2.] 2012 - Mendix WebModeler v0.1
    • [added Oct 5.] 2012 - mxplient
    • [added Oct 6.] 2012 - Access2Mendix
    • [added Oct 9.] 2013 - Stacky Bird
    • [added Oct 3.] 2014 - Sandboxes
    • [added Oct 2.] 2014 - cf-mendix-buildpack
    • [added Oct 5.] 2015 - certinator
    • [added Nov 28.] 2015 - buildpacks for Docker
    • [added Oct 3.] 2015 - InstaDeploy
    • [added Oct 5.] 2015 - Mendix2Java
    • [added Jul 25] 2016 - Mendix Cloud v4 - Part 1 - How it got started
    • [added Oct 9.] 2016 - Packing all mxruntimes in git
    • [added Oct 5.] 2016 - The too clever scheduling service
    • [added Oct 2.] 2017 - The Reaper
  • Part 4 - CTO @ easee ('18-'23)
    • [added Oct 2.] 2018 - The easee Wizard Engine
    • [added Oct 10.] 2019 - The easee Diff Viewer
    • [added Oct 11.] 2019 - easee RemoteControl
    • [added Oct 4.] 2020 - easee TeleVisus
    • [added Oct 22.] 2020 - QMS Integrity Checker
    • [added March 3 2024] 2021 - The WaterRower Camera
    • [added Oct 30.] 2021 - FDA Crawler
    • [added Oct 11.] 2022 - Cowboy e-Bike Remote Control

Part 1 - Fun but somewhat forced programming ('96-'00)

1996 - JTJ Rekenen

Niels, my dad, had a 13-year career at HP (later Agilent). He started servicing mass spectrometers and ended up as a programmer. He always loved computers. He was pushing me to get into software from about age 8 or 9. We started developing a small program together that taught kids how to do maths. My dad did most of the work, and we created this together with my friend Joscha. We called it JTJ Rekenen. JTJ stood for "Jouke-Thiemo Joscha", Rekenen is Arithmatic. We contemplated calling it JJT but JTJ sounded cooler. The program did simple addition, multiplication etc. In hindsight, I believe the logic for division was full of problems.

The UI let you practice +, -, * and /, and turned the screen green if the answer was good and red if bad. I sold this little program to my school for about 50 guilders. I was pretty proud, but in reality my dad made most of the software while my friend and I watched and did some simple things and nodded a lot. JTJ Rekenen was built in Delphi or Visual Basic. I forgot which, but we used both at this time.

1997 - Plotter

As my dad worked at HP he saw all kinds of equipment in the office. At some point he brought home a couple of broken inkjet printers. He took out a lot of parts, ordered a microcontroller from the US and created a simple plotter for A4 paper with the stepper motors. Back then microcontrollers were expensive and difficult to program, there was no Arduino IDE or anything like it. If you weren't careful you could blow out your parallel port.

Dad built all the hardware, connected the microcontroller, and hooked up a driver and a small library so we could control it from Delphi. My job was to program the code to move the head around and up and down. Up means the marker is not touching the paper, down means you're drawing. This was great, as you could program something and see the results in the real world right away! I still enjoy try to get instant feedback loops whenever I'm working on a system.

I started with creating the logic for the letters. If you draw an A, you had to create 3 lines, 2 diagonal and 1 horizontal. On and on for the letters B to Z. It was quite a lot of work but after some time I got good at it and start optimizing patterns. Dad showed me how to make procedures to re-use logic. I figured out how to abstract things so that letters could be drawn from a relative starting position, how to do line wrapping, etc. Lots of fun.

1999 - Reverse engineering the binary format of HP ChemStation

At this time of my life I did not program for fun, I did it because my dad asked me to. I did a lot of other things with computers though, mostly games and drawing (CorelDraw). But in the summer of '99, my dad was struggling with the software that Agilent used for their mass spectrometers, ChemStation. It was a pretty big software suite and development was done in California. He could not influence it. He was often frustrated with how it worked, and like a true hacker, wanted to read and manipulate the data files with his custom software. He still does that to this day, now mostly with ChemStation's successor MassHunter. The ChemStation data format was super optimized for storage space and thus hard to understand. It stored the raw data from the spectrometers and some data that was annotated by the analysts in the lab.

On Friday nights dad, me and my two younger brothers used to go swimming in Amstelveen next to the HP office. One night in the car on the way back, he said that he could use my help for a couple of weeks during the summer break and could pay me about 100 guilders for it. He wanted me to figure out the data format that ChemStation used, so that he could write a parser for it. He showed me how to manipulate the data in ChemStation, and then inspect the files with a hex editor. If I made a copy of the file and changed the data I could compare the before & after versions and see how things were stored. Wow.

I learned so much in these couple of weeks. I think this was the first time I actually created something useful for someone. He could have done it 10x quicker himself, but what a fantastic learning opportunity for me. I learned about little/big-endianness, hex/binary/string encoding, and variable length fields.  The result was a pretty good description of the data format. When my dad implemented the library he would sometimes call me over to explain some details that he didn't understand. I felt awesome and was only 11 years old!

In 2010 I worked for him for a year, and one of my projects was to create a C# library for the same old ChemStation data format. I think he still uses it to this day.

Part 2 - High School and Uni ('00-'11)

2004 - The HP 49g

In secondary school, everyone that did "advanced" maths had to get a TI 83+ calculator. Being an "HP family", my dad bought me an HP 49g+ calculator with RPN. I loved that thing. Going against the grain and doing things differently is very important to me. Here I had to get special permission from school to use this device because it was not the approved crappy TI calculator. Luckily I had a really cool maths teacher (hello Mr. Pieke!) and he made it happen.

I loved RPN and stack-based programming. I created all kinds of small programs in User RPL. On holidays I would be programming stuff in my tent. On hpcalc.org I downloaded all kinds of knowledge bases for e.g. the periodic table and loved reading things from the calculator. The calculator broke down at some point, then I bought a 49g+, but that was stolen from my student room later. I still miss it.

2005 - Porting an Atari tax program to Delphi

My mom made me take side-jobs from age 13 and that's something that I'm really grateful for. It gave me a good work ethic and plenty of money for a kid, but I always spent it right away and was never able to save any. I started delivering newspapers, and then graduated to baking bread at the Jumbo supermarket. I got fired after a year because I overslept one too many times, and had to find something new.

My friend Marijke from school had a neighbor, George, who was 18 and was starting a computer repair business. He was entrepreneurial and studied next to this full-time job. He also played in a band. He had so much energy. She recommended me to him because I was doing things with computers all the time. I was 15. The pay was terrible but I loved the work. I started with 4 hours per week, but by the time I was 18 was often doing 20-25 hours per week after school and in weekends. We started in his mom's house, and she would feed me Mexican food all the time. She was very sweet and I was way too skinny. Then we moved to a real store in the center of Middelburg and I had to arrange my own food.

I could program, repair and assemble computers. We had two more employees, Lotte and Bart, who knew very little about computers and I had to teach them almost everything. The assembling was the most prestigious work. The repair was excruciating, especially when the hard drives were semi-broken and de-fragmenting would take ages. Or when there were nasty viruses. People came in with "very important data" that they could not afford to lose. That was mostly porn.

There were so many mistakes (by me) that I don't know where to start. I lost computers under repair because we had no labeling system. Customers came in for weeks asking if their PC was done. I had no idea which one it was and had to play dumb. I was also very introverted and had to practice calling someone after a finished repair. Lotte and Bart were much better with those kind of things. Often I forgot the mainboard backplate when assembling the machine. Then I had to take everything apart and start over. I learned from my mistakes and became quite competent. I programmed our webshop, created a labeling system and built a couple of high-end PCs for rich customers.

Then one day, George came in with a big stack of paper. George was a great networker and sales guy, and he had made a deal. The paper was the source code for an administration & tax program for a local store. The store owner was an old geek and had built his own software in Basic(?) in the 80s. The Atari it ran on died and he needed a replacement on Windows fast. George put me to work. I had to decipher the code but I did not know the language or how a terminal worked. I did not have a working version of the software or a computer to run it on, so had to reconstruct it in my head.

I had to program the replacement in Delphi but I was also a complete beginner in that. It took me weeks to figure it out, and I'm not sure if George ever made money of the project. In 2012 it was still running, because the VAT rate in NL went from 19% to 21% and George contacted me to see if could update it. I did not have the source code anymore. Oops.

I learned a lot from George and from all of the mistakes I made. Thanks for the fun and the opportunity BG!

2009 - SnelTrein

While I was off studying Computer Science the smartphone revolution kicked off. Being an Apple hater, I got the first Android phone I could get my hands on: the T-mobile G1. Loved it and still think killing physical foldable keyboards was a shame. I had all kinds of ideas for apps, for some reason most of them revolved around trains. Like a GPS alarm clock that would wake you up when you got close to your destination.

I traveled by train a lot, and planning a trip or checking train times was a pain. I decided to build a train times app, but the name Trein was already taken, so I named mine SnelTrein. The Dutch name for high-speed trains. It was a simple wrapper around the mobile website of the NS (Dutch Railways). My app was super fast and super simple in the UI, as it remembered your trips. Your most often used trips were always on top. Once the app opened, it was literally one click to see the departure times of any of your 5 most used trips. I loved it, users loved it, and it grew to 3000 installs.

After some time, the NS released their own app, and I figured they are probably not scraping their own mobile site. After MITM'ing the traffic, it turned out they indeed had an XML based API which was password protected but they did not use HTTPS. I created a blog post post about this, submitted it to Dutch tech website tweakers.net. It was posted to the front page and I got my 15 minutes of Dutch tech fame. https://tweakers.net/nieuws/73527/ontwikkelaar-verkrijgt-toegang-tot-ns-api-voor-treintijden.html

I stopped maintaining SnelTrein because the NS app caught up a bit and is good enough. Early in 2022 I also switched to an iPhone SE from 2016 because Android phones are so big.

2010 Snapp

Stephanie's uncle Rob had a small design / marketing agency in Amersfoort, together with Bart. They contacted me to create a web-based "company personality test". A bit like 16 personalities / MBTI but then for companies. I liked the idea and could use the money, so I went ahead and said yes. The basic concept was to get companies back in touch with their core to brand themselves more effectively. E.g. you might discover that your company is a true "pioneer" or "peace keeper". If you're in touch with your personality, it should give you a sort of compass for your actions and for how to communicate more effectively. It was about 12 questions and you had to drag and drop your 3 favorite answers per question. Then you got 1 of 12 different personalities at the end. After the test they could help you create a brand strategy and implement it.

About a year or two later Rob & Bart needed a similar website with a quiz, and they asked if I could create the same thing again. It was using crappy jQuery technology so I wasn't too excited about it. The Dont-Repeat-Yourself mantra kicked in, I had a bit of a feeling that there was a cool tech solution and decided to create an engine and editor for creating quizzes. This turned out to be a really cool engineering challenge. I went a bit wild on my own time and within months you could:

  • create quizzes in diagrams with a drag & drop editor interface
  • embed it on any website
  • see live traffic as people were taking the quiz
  • have persistent data stored in mongodb
  • have non-linear presentations with paths dependent on previous answers
  • have a pub-quiz-like interface for many users

Slowly but surely it was becoming a full-fledged programming language. I experimented with different names, and went through Snap, Snapp, SnapLogic, WebFlow, SurveyMaker. Most of these were taken by similar companies though. Rob & Bart were very impressed and really loved the direction this product was going in. We started talking about creating a company together. I would do tech, and they would find customers and do implementations. However, by the time that it became really serious it was mid 2013. I was very busy with my day job, getting a kid and interviewing at facebook. I couldn't afford to go all-in as we had no funding and I was pretty much living paycheck to paycheck. Besides, the tech at Mendix was a lot more exciting than this.

In conclusion Snapp was just too low on my list of priorities and it slowly died after this. In hindsight there was a lot of potential. I had a great time with my career choices and don't have any regrets, but still I sometimes wonder "what if" about this venture. We'll never know :)

Part 3 - The Mendix Years ('11-'18)

2011 - newnode.py

At Utrecht University I had done a dual BSc. degree in Computer Science and Cognitive Artificial Intelligence. Most if it was technical but I had done plenty of courses in the humanities as well. Some psychology, history of philosophy and Arabic. I was also lucky enough to have joined a book club where we read the classics. I made great friends there. One of them invited me to an honours program in Rhetoric in Amsterdam. That class had about 30 students, and Alexander Klöpping was one of them. I remember that he flew to NYC to get the first iPad and then talked about it on national television in "De Wereld Draait Door". He was in a very different world from all the other students and it was really cool to see. 

When I started working life I knew I was going to miss the humanities. I registered for an evening program for a BA in English Language & Culture at Leiden University, My first job was with my dad, and it didn't really work out. It was just me and him and I struggled with motivation or even to get out of bed. People in my English classes saw I wasn't happy. Being slightly alcoholic, I joined a gang that went out to a bar every Thursday after class. Annemarie, Maryssa and prof. Liebregts were there most of the time and we had lots of fun.

One of the assignments for class was to write a job application in English. Annemarie wasn't happy in her job either and had met Derek, the CEO at this tech startup at a party of a friend. He was hiring sales people. She decided to write an actual application and got hired. Within a couple of weeks she told me that they were looking for developers too. It sounded cool and she introduced me to Johan, the CTO. After an email exchange I visited the old Mendix office on the Westzeedijk. I didn't realize it at the time, but actually Johan had only taken over the role from Roald a few months before. I think I was one of the first hires he did as CTO. Working at Mendix was an experience for which I will be forever grateful to everyone there, and a special mention to Annemarie for suggesting it to me. I ended up staying close to 7 years in various positions. 

Because I was running Ubuntu on my computer, Johan said, "oh Linux, just like the cloud guys. Let's put you in that team." I got hired, but actually I didn't know anything about cloud, servers, or even Linux. I just really didn't like Microsoft and Apple. 

We had two "cloud" teams. One was the SysOps team that was responsible for customer hosting & internal IT. The guys there were extremely good system engineers, and had built a rock-solid hosting operation in two colocation data centers in NL. The CEO didn't like it. By 2011 the cloud was the cool new thing and servers in a data center were obviously not the cloud. You know the old saying: If the CEO says Cloud, you just say "which one". They had looked at AWS and some others and had decided on Linode. It was more affordable and their culture fitted well with the hacker ethos of Hans, the senior systems engineer. 

As the 3-headed SysOps team was already drowning in work, a new team was formed. This was Arjen and Achiel, and later myself. Once I joined, Arjen went back to Runtime, his job in the cloud team was done. Achiel and I had to build the Cloud, while the SysOps team would be maintaining the serious production systems.

I loved working with Achiel, he was so much fun. We had Mario stickers on the wall and were laughing all the time. We had a running joke that we could only use Dutch software to build the cloud: Python, Mendix and vim. He is a great engineer and now PM at CloudFlare, and he taught me so much. But if we're honest we both knew very little about systems and hosting. We were both just regular developers, he senior, I junior. In the cloud team we had to build productions systems and work with a lot of things out of our comfort zone:
  • Hosting Reverse Proxies, JVM based apps, Databases
  • Networking protocols: DNS, TCP/IP, HTTP, SSL/TLS, XMPP, Load Balancers, NTP
  • Servers: RAM, CPU, hard drives
  • Services: Backups, Monitoring, Alerting
  • Distributed Systems and operating them 24/7
  • Scripting in Bash, Python etc.
It was a lot. I would say that on a 0-10 scale, the job required a skill level of 6 on all these, and I was a 1, maybe 2. Achiel was a bit ahead but we really needed the SysOps guys for a lot of guidance. By now I think I'm at 8 for most of them. Over time everything went wrong so often that I simply had to learn how it worked.

Within a couple of weeks of me joining, Johan took a holiday and Roald started giving us feedback in a sprint review. Johan was always nice, but Roald gave us so much (justified) crap that I will remember that meeting for the rest of my life. It was a heated discussion and having recently joined I only observed. I still got a bit scared. It took me years to make a 180 and now I sometimes wish I was as clear and as passionate about user needs as Roald.

My first task was to automate the creation of new cloud VMs. We had inherited a wiki page with instructions on how to install a virtual machine according to Hans' specifications. We would need at least 100s of VMs, and indeed, a couple of years later we had 6000+. Being a good lazy developer, I wanted to automate this. I created `newnode.py` which would spin up two new VMs on Linode (1 for app, 1 for database) and set up the infrastructure to run one Mendix app. We had a naming scheme for different data centers. Linode had Japanese dishes (appnode-unadon), Rackspace US: Star Wars (appnode-aldebaraan), Rackspace UK: Star Trek (appnode-stargazer).

The script touched all the services & protocols above. The things that could not be properly implemented were using `expect`. I learned about "when using 'expect', expect trouble" the hard way. Despite my lack of knowledge with any of the Ops stuff, and being very new to Python, it was finished within a couple of weeks (or months?). I had this big smirk on my face, because I could do in a couple of minutes what the Ops guys were doing manually in an hour. Now to be fair, the script outputted a lot of information that we had to manually put in our DNS servers and other systems which only the SysOps guys had access to. It still took some years to automate everything end-to-end. Also, setting up a server is easy, maintaining it over time is hard. Nevertheless, it was a great step.

Hans was very much against going to the Cloud in the way that we were doing it. A bit of Not-Invented-Here syndrome maybe, but also because of costs. He could run a more stable operation for about 5x less money because the hardware he set up was tailored to exactly our workloads. It helped that he probably forgot to include his own salary in the calculation, but it was still a lot cheaper ;) After a couple of years we ended up running the same kind of VMs on our own infrastructure in three different Data Centers in NL. It was the financially sensible thing to do. In the US we kept running on cloud VMs because we had no boots on the ground. In both setups we kept using newnode.py or an evolution of it. Hans' vision was always that if you go cloud, you should really go cloud-native instead of running the same old "One VM per app" approach. And until we would have that, we should stick with hardware. In that he was completely right. It would take years before we could get to that next generation infrastructure.

The SysOps and Dev spirits kept clashing a bit over the years, until we decided to join forces and become the DevOps team together. Achiel and Johan had read "The Phoenix project" and were sold. We took the best of both worlds and it was great, and I think everyone on the team is still nostalgic for that special time in our lives. We won the Team Excellence award for it a couple years later.

2011 - Social Gravity

I got onto twitter in 2010 and at some point realized the immense knowledge stored in the network. If you know who someone follows, and who follows that user, it tells you a lot.

With javascript webworkers I created Social Gravity. A 2d visualization of someone's network, all followers and their relationships. It had a small gravity engine, where you push away everyone by default, but if you follow someone, you attract them. If users get closer you repel them exponentially, so that the network does not collapse into a single point.

Magically little groups of your different spheres of influence or interests appeared. If you hovered over an area, it highlighted frequent keywords in the bios of the twitter accounts. You can try to market yourself in any way you like, but your network doesn't lie.

As twitter restricted their APIs a lot, and I was very busy with my cool new day job at Mendix, this project died down. I created a video with how it worked here: https://www.youtube.com/watch?v=pyVUMTVarpQ . It's a bit embarrassing because it tells you so much about my interests back then. Anyway, most social network APIs don't expose this data anymore, but they use this data for all kinds of advertisement targeting. Pretty scary.

2012 - autopullpep8 and the facebook adventure

At Mendix, Achiel, my first team lead showed me Python and it became my go-to language. I got frustrated with the different coding styles, whitespace errors, and learned about pep8, flake, etc. Nowadays we use `black` but back then there was only `autopep8`. A tool that reformatted python code to the pep8 standard.

Learning about GitHub, its API and autopep8 I got a brilliant idea. What if I created a script to clone the top 100 most starred python projects, applied autopep8 and submitted a Pull Request. All completely automated. This was simple. I was nice enough to check the diffs before sending the PR. Autopep8 got a lot wrong and made some things really ugly. I spent 20 hours cleaning up code and then submitting PRs. I got my (small) contributions into a handful of top python projects. Pretty cool! Now more people figured out this trick and it is frowned upon in the open source community, but I think I did more good than harm.

I will never know for sure, but I suspect getting patches into top python libraries triggered some recruitment algorithm in silicon valley. In mid 2013 I got a message from a recruiter at Facebook for a position in their Production Engineering team. I actually hated Facebook and had deleted my account a few months prior. On the other hand the job was super cool and I loved California. I thought I had 0 chance of making the interview but it might be a great learning exercise. So, the process started. Over a period of 2 months I had 5 late night calls with the US West Coast that I had to keep secret from my bosses. I didn't prepare much and had the feeling that I passed every interview by the skin of my teeth. 

My brain works a bit strangely. I can't grasp theoretical knowledge well, so while studying Computer Science a lot of concepts did not stick. But in these interviews, I finally understood what the engineers were struggling with and how we could create solutions. The theory finally had a useful application! I learned so much during and after these calls, and I'm super grateful for the experience. 

At this time I was offered an "incentive package" at Mendix because of all the projects I did in my free time. I felt obliged to come clean and told Johan about my "affair" with Facebook, and that I appreciated the effort but first had some choices to make. I think the news spread quickly and most people considered me a lost cause. Facebook flew me out to Palo Alto for two days of on-site interviews. I did a bit of a road trip with my dad who by pure chance was freelancing in LA. He came over to the Bay Area for 2 days, we rented a fancy car and had fun.

The Valley was cool. While at 1 Hacker Way I saw Sheryl Sandberg doing a walking meeting across the FB campus with 5 of her reports. When I checked into the Sheraton there was a guy in an Instagram shirt standing next to me, he looked very happy. Instagram was bought for $1B a couple of months prior and the guy was now worth north of $50M but he still looked pretty normal. This was a different world. I also saw my first Teslas on the road and was stupid enough to put my actual current salary on some HR form.

Back home I got a good offer, negotiated a bit and got a better offer. IIRC it was 128k + 2x yearly 20k signing + stock, and for a 25-year-old engineer from Europe back in 2013 that was amazing. I still don't make close to this amount now, even though I probably could ;) In the Valley it was probably "meh" and that's why they recruited from overseas. But remember, I never expected to get an offer because in my head I wasn't good enough for the Champions League of the tech industry. So now I had to think. While thinking I discovered that:

  1. I don't care about money beyond basic needs.
  2. I'm very loyal and liked our mission at Mendix a lot.
  3. I care about the environment. Flying back to Europe 2 times a year with Stephanie my wife and a brand-new kid did not sit well with me. We now have 3 kids and it still doesn't.
  4. I don't like joining a winning team, I like a challenge and have a giant weak spot for the underdog.
  5. I'm actually a pretty great engineer.
  6. The H1B visa would not allow Stephanie to do any paid work. The H1B spouse club is real and that's a pity.
  7. I want to work on something good and meaningful and still hate Facebook.

I declined the offer. I felt bad about the super smart engineers and the recruiter for going through all this trouble (sorry Phil). I had made it clear from the start that declining was a very real opportunity. Thank you Zuck for sponsoring the free trip and the great learning experience. At Mendix, Roald told me he was very relieved I decided to stay. That was nice.

2012 - JUDO

At Mendix I worked on the Cloud team, we hosted Mendix apps for customers. Hosting should be as easy as building. The Cloud/Ops/DevOps bunch was a tightly-knit team, as we were in the firing line together. With that I mean 24/7/365 uptime guarantees and you need to rely completely on your colleagues at all times of the day. It's something else. When I joined we were 4, when I left it was 12. Now it's 100. I'm in touch with a lot of these people to this day.

Every night the system created backups of the apps which admins could download over HTTP. The download consisted of the postgres dump, but also a lot of files from object storage. We could store ZIPs for every backup every night, but then the storage requirements would explode. Instead, we did incremental backups with rsync and hardlinks. Achiel wrote his BSc. thesis on this project for the sole reason of graduating within 10 years of starting college. If not, he would have to pay back his government study subsidies. That was fun. Hans was actually the mastermind behind the backup system. He's an incredible engineer and always built rock-solid stuff. Now this system was pretty cool, but the download needed to be a single zip file and that was difficult.

We built JUDO, the Java Upload Download Organizer. When getting a download request over HTTP, it assembled the zip file on-the-fly while walking over the files storage directories. This was a pretty complicated project and Xiwen, Frank and I needed a lot of help from the senior engineers (Hans and Achiel). Over the years we found obscure bugs and fixed most.

The streaming approach was very elegant. There was no temporary storage and no asynchronous processes. The fundamental problem was that the size of the file was not known at the start. The size and Content-Length header could not be predicted because of the compression. When a connection got interrupted, browsers were not sure whether the download was complete. This led to some problems. This is why most products use a "we are preparing your download" approach. Less efficient, but when dealing with backups it's important to know you have downloaded it completely.

2012 Decrypt.py

With my new python skills I created a funny little program called decrypt.py. I had done some hard disk recovery for a friend, and found it a bit ironic that the end result was so bland. It just said "19783 files recovered". In movies it's always way more dramatic. My creation brings the drama back to boring things. It takes input, shows garbled text on the screen and slowly reveals the original content. You can view a demo on the github page.

It only took a couple of hours to build, but the main thing about decrypt.py is that it got me really hooked on Hacker News. I had discovered it a couple of months before, but had never posted anything. Now with this post I reached the top spot for a couple of hours (I think?) and saw the comments and upvotes coming in. It was very addictive. I've reached the top place a couple of times now and it's a rush every time. There have been very few days that I have not visited the orange website since 2012, and I did learn a lot from it.

2012 - Mendix WebModeler v0.1

Mendix was (and is) a fantastic platform. Most Mendix developers use the Desktop Modeler / Mendix Studio Pro to create apps. Back in 2013 this was the only option. I had a different vision for the product. Maybe because I had no love for Windows apps, maybe because my stack was the browser, maybe because of the actual user needs. In any case, I wanted a web-based app creator. No installers, no friction. It would need a lot of difficult front-end engineering and a complete revamp of the codebase.  Think of how much effort Microsoft had to put in Visual Studio Code. Add another re-architecture for client-server and you get the idea.  No one within Mendix was crazy enough to try it.

Being young and unhindered by wisdom or experience, I spent a couple of nights on creating v0.1. It was a PHP + jQuery based application that could view the microflows and models for a Mendix project. This logic comes from an MPR file. This is/was a SQLite database that contains all modules, logic and forms. I loaded the Mendix Cloud MPR file, put it on a password protected server and sent the link around to the CTO, CEO and some other people.

I remember watching the live server logs, seeing them play with the prototype. At some point I saw an IP from the US. That was Derek, the CEO. Boy that was exciting. I had a chat with him a couple of weeks later, and if memory serves, he said something like this. "The web modeler is clearly the future. We need this but more urgently we need something like a sandbox environment so people can try Mendix for free and show their apps to their colleagues." We got there too, but that's a story for later.

About a year later we started a dedicated team for the Web Modeler. I was not in there, neither was any code from my prototype, but I like to think I helped shape the vision and showed that it was possible to bring Mendix to the web.

The Web Modeler needed a lot of serious front-end engineering indeed. Michel Weststrate was one of the main engineers and he created MobX (then Mobservable) in the process. It took years to mature, but nowadays the Web Modeler / Mendix Studio is a serious part of the Mendix ecosystem and I'm proud to have been there.

2012 - mxplient

The Mendix core had three parts:
- The MxModeler (IDE)
- The MxClient (Powering the app in the browser)
- The MxRuntime (Powering the back-end and managing the database)

I was in the Cloud team and used a Mendix app for the Cloud Portal. As we needed to automate a lot with external scripts, I needed an API in the app. Mendix had no REST functionality yet, and being Python fanatics we didn't want to touch XML WebServices.

Then I realized the app already had an API. In the browser UI, I was doing all these actions I needed to automate, and those actions went to the Runtime as HTTP requests. Using the browser debugger I discovered the undocumented API that the MxClient and MxRuntime used. I proposed just making this API public, but the Mendix core teams were very against this. Who did this junior engineer think he was? Michiel Kalkman was the main engineer of the MxClient at this time. He explained that this was a proprietary API that could change with every Mendix release. Not that that happened, but it could. Theoretically. Every couple of years. He was an extremely good senior engineer who had a lot of ideas for improvements to the API. If you're a visionary, you can get disappointed because not all of your dreams come true, or take longer than expected. In practice the API was very stable.

In the long term he was right of course, as some big releases had big changes to the protocol, but that didn't stop me. For our own use I created the "mxplient" library from the reverse engineered protocol. The P stood for Python. I used it for some small scripts, like newnode.py, or whenever I was playing around with something. One by one people started discovering the project. It was a tiny library, badly designed, and I had no idea with which Mendix versions it could work with. I think Michiel kind of appreciated the hacky way I did things. He liked Python too and had built a similar project in Ruby. I think it's the only Ruby code that was ever seen at Mendix.

Around this time we had hired Daria (QA Engineer) whose job was to test the Cloud Portal. I once showed her the script. I blinked and then one year later she had built a giant automated test framework on mxplient. Oops. Then I felt guilty and started improving it. I don't know if it's still used, but this was a nice project and became my go-to solution if I needed to automate something with Mendix.

Also around 2012, Michel Weststrate built the REST-Services module to provide a more standard and stable API to users. Years later, REST was properly added to the Mendix core itself. It's funny, Michel and I were both very productive hackers but we always had a different approach. Michel would engineer solutions as Mendix add-ons (taking quite some shortcuts). I would reverse-engineer and create workarounds completely outside of the Mendix ecosystem. Much later the "serious software engineers" picked up the pieces and made real products out of them. As Michel's work was available to the entire Mendix community, his work had a lot more traction and most got included in Mendix itself at some point.

2012 - Access2mendix

I'm not sure how this project got handed to me. Maybe we were a bit desperate to find new revenue, and in that search, someone had realized that a lot of enterprises used mini-apps built on Microsoft Access. Microsoft Access was installed on most systems because it was bundled with Office, and it allowed mildly technical users to build apps. Going through proper IT processes involved layers of red-tape so lots of people had built lots of solutions with Access. As I hated red-tape too I understood those people very well. They were what was called "shadow-IT". These apps were stored on some shared folder which only a couple of people knew about, and no one knew how to maintain it. Lots of processes were going through these "unofficial" apps and this was seen as a problem, because IT was not in control.

With Mendix we thought we could do better. Our platform was more secure, more scalable and supported smartphones too. With our centralized admin tools, IT would be in control.

We would target these companies directly and try to sell our platform. By this time Derek and Martijn had moved to the US and had started our Boston office. To blow US prospects away, they needed a Demo with capital D. Martijn was already amazing at demoeing Mendix, and built apps while the potential clients watched. It really was something to see. Here I learned that "the product will sell itself" is BS. You need great sales people, and at Mendix we had lots of them.

To tailor the Demo to their new story, it would be great if they'd have a customer upload their old Access app, and get a Mendix app out of it within minutes. This somehow ended up with me. I believe Derek remembered my WebModeler v0.1 experiment and bypassing two layers of management, asked me if I could take a look at this "Access2Mendix" solution. CEOs will be CEOs I guess.

What followed was the most productive weekend I have ever had. Normally working weekends is crap, there is a deadline and you don't really want to be working but you have to. Here there was no deadline, no pressure, just a very interesting challenge.

It would be really powerful if Access2Mendix was built within Mendix, so that's what I created. It was 99% Java code, but no one would know that. I wanted to convert three things from the Access app:
  • the data model
  • the data
  • the logic
The forms were kind of impossible to convert, so to compensate for that I created default admin forms, one per entity in the data model.

You have to understand that although I was in R&D, I was a complete outsider to how the Mendix model worked. I used Mendix, but had no insights into the specification. In fact, there was no documented specification. It would take another couple of years before Meinte and Jos would create that. I had to reverse engineer everything.

First I learned how to read an Access file using an open source Java library. Then I took an "example project" from Mendix and started manipulating the MPR file manually, inserting new modules, models, microflows and forms. I believe I was the first person to ever attempt this, and the Modeler crashed in weird ways whenever I made an error. The development cycle was super addictive. Add some conversion logic, upload, see the result within seconds. If it worked, go to the next functionality, if not, fix the conversion logic.

I literally worked from Friday evening 7pm till Monday morning 8am and only slept 6 hours per night. At the end I had a Mendix app where you could upload an Access file and within seconds you got a fully functioning Mendix app + database back. Customers loved seeing their actual data directly in the new app. It made for a super impressive Demo.

Unfortunately the whole Access conversion market was less spectacular than we thought and the project was abandoned after a couple of months.

The main thing I got out of it was a reputation within our sales and management teams. In turn I gained a lot of respect for our sales people and made some new buddies within the company. A couple months later we had the first Company Kick-Off at Duinrell. At the party Derek started handing out awards. It was great to see each other's contributions and celebrate them. During his short speech towards the first ever R&D Excellence Award I suddenly realized, oh crap, he's talking about me! I had to get on stage and after the initial rush of introverted feelings I was very proud. I believe this project was one of the contributors to me getting that award.

2013 - Stackybird

The year 2013 was tumultuous. We were doing lots of projects at work, we moved to a fancier office on the Gedempte Zalmhaven, and I was secretly interviewing at Facebook. My personal life was about to be upended too, as we figured out that Stephanie was pregnant in February/March. Coming home one Friday night there was a surprise pregnancy test on my desk. Two thoughts fired like rocket ships in my brain, in opposite directions. One was happy and excited, the other scared of the responsibility and the stress. My facial expression was switching between them for a couple of seconds. Then I realized the nice thing to do for Stephanie was to focus on the first rocket. She must have seen the hint of horror in my eyes though, and it makes sense. Any good parent should be at least a bit afraid about having kids and it is a giant responsibility to put them on this earth.

Hugo was born in October 2013, and two weeks later I was flying to SFO for the interview at facebook. Thanks for allowing that, Stephanie. 

A small part of my responsibility as a parent was to give my new son the opportunity to learn programming. I had received that gift from my father, and I wanted to pass it to the next generation too. When he was a couple months old, I went and created Stacky Bird. A bit premature, but it was on my mind already. I expected him to learn coding around age 8. You can see Stacky Bird here: https://game.stackybird.com .

Stacky Bird was a combination of three of my interests:
  • teaching my kids programming
  • creating a programming language from scratch
  • the "meme-games" Flappy Bird, 2048 and SpaceChem that were popular in 2013/2014.
The game allows you to solve problems by giving a bird directions. The initial version was super simple. After some time I realized that it was actually a programming language, and with a bit more features is probably Turing-complete. We got two more kids, Suzanne and Louise, twins, in September 2015. When the kids were 7, 5 & 5 I did a thorough rewrite of Stacky Bird in VueJS and they could actually play it. They liked the minecraft coding on https://code.org a lot better though.

I'm not a helicopter parent who's hyper-involved with the kids, but still, kids take up a lot of time. My energy levels dropped tremendously and it would take years before I got close to the 2011/2012 levels of productivity and energy. The hobby projects, hackathons and weekends of programming disappeared for 90%. Most of the projects you see in this post were done during working hours. I felt held back compared to colleagues who were able to power through weekends, especially in the startup I went to in 2018. Now, in 2022 I finally feel like I am getting back in the game. Sometimes when I have kids duty, I nap on the couch and my kids run around the neighborhood. So much freedom compared to the diaper phase!

I was 25 when I got my first kid, and 27 when I got the last two. My advice for past-me would be to start a year or two later, but not much more. I love being a young parent and if the kids leave the house around their 18th, I'll be only 45. Again there's two thoughts. I'm looking forward to the freedom and travelling with Stephanie, but I'm also scared of the empty nest.

2014 - Sandboxes

Derek had a dream for the Mendix trial and he kept talking about it. Sandboxes. You could already try Mendix for free, create an app and run it on your own PC. But you couldn't share your apps to your colleagues. We needed a trial environment in the cloud. Derek was an excellent CEO because he had a clear vision which he kept repeating. He said what users needed and didn't micromanage the solutions. For all I know the higher-ups might have experienced a different side of him, but being at the bottom of the food chain this was my perspective. I have tremendous respect for him.

We could not have thousands of free apps running on our infrastructure, costs would go through the roof. Serverless didn't exist and the Mendix runtime was quite heavy. The only remaining solution was what Heroku did: stop the app when not active and resume it when traffic came in. I had learned a lot while building the Mendix Cloud and during the interview process at Facebook. Armed with this knowledge and Derek's vision, I had a spark and realized how we could copy Heroku's feature. We could send nginx traffic to a mini app that would start up the main app. When started we would rewrite the nginx config to send traffic back to the main app and trigger a reload. I spent the weekend thinking and tinkering, and at the end I had a prototype of an app that could be resumed when traffic came in. Stephanie didn't see much of me that weekend. It was awesome.

On Monday I was like an overly enthusiastic kid and told everyone that we had the final building block for a solution. Somehow it got on the roadmap. I had no idea how that roadmap process worked, to me it was all energy, inspiration and magic. Actually that building block was just one tiny step for the Sandboxes release. Because it was the only "magic" part I focused on that, but there was so much more. We still needed:
  • Cloud infrastructure that could support dynamic re-assignment of apps to virtual machines.
  • A "last request" counter in the runtime.
  • A deploy button in the Desktop Modeler.
  • A "your app is resuming" page.
  • A runtime license for the free apps.
  • An SSO service for inviting your colleagues.
  • MxBuild, a cloud based compiler for the apps (not sure why it was needed, but we did build it around this time).
The MxBuild project was interesting, Benny and I had built it on a research day once, and it was the best pair programming experience I've ever had. He was building the Mono executable, and I built the python code that called it. For some reason we were so well attuned to each other, that we would see the other side of the API being typed on the other person's screen, and typed the other side on our own screen. An error popped up on my PC and Benny would fix it on his without a word. With hardly any explicit communication the entire thing just spontaneously came into existence. Two minds working as one, it was really beautiful.

But back to the sandboxes release. We also needed more hardware! The co-location data center in Amsterdam was getting full so we needed to add a couple more servers with a lot of memory. For that we needed the CFO to sign off on a couple $100k in infrastructure costs. That was difficult because we knew we were going to the Cloud soon. Derek had been saying it for years. Why did we need more money now for hardware? Well, the "real cloud' would not be ready for about two more years. Thanks to Hans' and Pim's engineering we had a super cost efficient infrastructure in co-location DCs. Bert (the Head of Finance) asked us some questions and explained some financial models to us, that was enlightening. We figured this was actually a great investment and AWS was going to be very expensive. Oh well. It would still be making us money.

It's pretty cool how we engineered this mega feature with effort from so many R&D teams. IIRC the launch went great, but quite quickly the VMs we used were filling up. Within a month or two I needed to hack a script to delete unused sandboxes. We kept doing that until the launch of Free Apps, about a year later. The infrastructure behind Sandboxes was a stopgap solution until we had Mendix Cloud v4.

2014 - cf-mendix-buildpack

The 12-factor app manifesto made an enormous impact on the world of cloud. If you didn't do 12-factor apps, you were stuck in the stone age. At Mendix we were stuck in the stone age, and we wanted out.

I believe it was in 2013 that Johan, the CTO, spent a weekend on getting Mendix to run on Heroku. He was still close enough to the Java code that he could compile his custom runtime for this purpose. I think this is the last serious programming I saw him do. He's a fantastic CTO and unlike me learned to let go of coding.

We never pursued Heroku again, but a year later Mark Rogers, who was leading Business Development, discovered Pivotal and their Cloud Foundry offering. This was an enterprise platform that targeted the same market as us. Cloud Foundry was Heroku for the Enterprise, and it was awesome. It would take us out of the stone age and into the 12-factor scalable future.

If I recall correctly, Johan asked me to try it out and to see if I could get Mendix to run on it. I was skeptical, but once I got access I gave it a spin. Being a Product Manager I now had a bit more freedom to spend my time than the daily scrum rhythm. I took the m2ee-tools code that Hans had written, wrote a tiny bit of glue code and hacked together a very ugly buildpack. Hans hated it because I took 12329 shortcuts. It worked though. You can see most of this history here: https://github.com/mendix/cf-mendix-buildpack . The initial commits were too ugly to keep so I rebased it. Initially it ran on both Heroku and Cloud Foundry.

"Production Ready" is a relative term. We were going all-in on the Cloud Foundry ecosystem and needed to run on IBM Bluemix, HP Helion and Pivotal Cloud Foundry. I think I became "mister cloud foundry" within Mendix, at least Erno started calling me that. Within a couple of months we flew to Barcelona for HP Discover, and I saw Johan on stage with Meg Whitman. He was talking to 5000 people about how easy it was to create a buildpack. "It only took us a couple of days!" He's a great showman, even with the Dutch accent ;)

The buildpack allowed us to go 12-factor. It needed major changes to the Mendix Runtime to remove state and scale horizontally which took about a year. Years later, in 2018, we had the first apps that really could not work without horizontal scaling. We were running an app that powered the "huishoudbeurs". A big event in the Netherlands that would see huge spikes in traffic. It ran on Cloud v4 and was very horizontally scalable. In my memory there were very few issues, and it was a proud moment for me.

The buildpack also allowed us to seriously partner with Pivotal, IBM, HP, SAP and GE. It gave us a seat at the table. Our Private and Public cloud propositions were much stronger with the Cloud Foundry play. Nowadays Cloud Foundry is not doing so well, and the Kubernetes beast became the standard platform that CF wanted to be. In my view CF has a much better developer experience than K8s and to me, that's what matters.

Although I was no longer an engineer but a Product Manager, the buildpack was clearly my project. It was small compared to the effort we put in m2ee-tools or the Mendix Runtime but it was crucial. It's my #1 project in terms of business impact. Every day, thousands of Mendix apps start running using my code. Although better engineers have probably refactored most of it, I'm immensely proud of it.

2015 - InstaDeploy

When we launched "Free Apps" I was pretty proud. It was basically "Sandboxes V2 + WebModeler". Under the hood we had this whole new architecture with all kinds of clever "cloud-scale" solutions. It turned out that users were not so happy. Every time you changed something in your app you had to deploy it. We wanted people to use Free Apps so we hid the old "local deployment" option. The Free Apps deployment service was buggy and slow. Local deployment used to take 10 seconds. The old sandboxes took 1 minute, the Free Apps took 4. On bad days it took 8.

The cloud team had launched the deployment service without any SLAs or basic explanation of the limitations to the other teams. A bit immature. The WebModeler and Modeler teams just had to use it and see how it worked. In testing it was already very flaky, but under production usage it was really bad. We thought we could get away with it because in the cloud, the rubber hits the road and we had failing stuff all the time. The other teams had stable unit tests etc. Not us. We thought that was cool. Now the ones who got most of the flak from management was the WebModeler team. They had all the visibility and the deploy button was in their product. They felt helpless.

Me and the cloud guys were not very open to helping them out. I was under a lot of stress to launch cloud v4. In the meantime we were scaling cloud v3 and needed new hardware every couple of months. This took most of my time and mental energy. I also could not justify the focus on our Free Tier while our paid production workloads were in danger. Sorry for having to deal with that, Erik vd P., Michel, Kishen, Daniel D. and Arjan B. . After it got really bad we took part of the blame and we started figuring out how to optimize the deployment time.

Then we had the annual Company Kick-Off in Noordwijk (the Mendix CKO). I ran into Derek, and said that I was confident we could get it to two minutes. His response was very Steve Jobs-like: "Two minutes? Why should it take any time at all? It should feel instant! If I make change I want to see it right away!". F*ck. He was right. We needed to rethink our entire approach.

I've never had a clearer call to action. I don't know how, but I managed to convince Johan that we had to put the top hackers from every team in one room for two days. This was Benny for the Modeler/MxBuild, Michel for the WebModeler, Xiwen and Daniel for Cloud, and maybe some more people. Let me know if you were there. I was there too, "managing". Because of Derek's speech I was fired up. Everyone else was happy we could finally fix the deployment problems. We all felt like we were on a mission from God.

We added some timing logs, broke down the deployment process and started optimizing our own separate parts of it. We started at 4 minutes and had ideas on how to get to 2. I realized we had to take ugly shortcuts to get beyond that. Instead of sending the deployment package to the deployment service, it had to go straight to the running container. This was very unholy and the 12-factor-app gods did not approve. I thought of Bill O'Reilly. "Fuck it, we'll do it live". We modified the buildpack to include a compiler on hot standby. When it got a deployment package on a separate secret HTTP path, it would trigger the compiler and reload the app. Benny built the mxbuild server. Xiwen got it to run in the container. Michel optimized the webmodeler so it could create a deployment package in 10 seconds instead of 20.

The time kept dropping. We were at 40 seconds. Benny optimized some more with pre-warming the build cache. Michel said "a-ha" and got package creation down to 0.1 seconds. At night Xiwen or me finally got mxbuild to run. On the second day the end-to-end unholy deployment cycle was working. Not in 2 minutes, not in 1 minute, but in 2 seconds. Holy shit.

It made for a great demo, a great story and when it worked, a great product. It never got super reliable, but boy was it fast.

2015 - mprtools

TODO

2015 - Mendix2java

Roald called me up one day and said we had a problem. We were about to close a major deal with a large company that I can't name here, but the sales process was stuck at the last stage. Roald was co-founder and pretty important and he never called me, so that was interesting.

He's an interesting character: driven, business focused, and very smart. He's unbeatable at the interplay of tech and sales. He's very confrontational, so I had to get used to him in the first couple of years, but after that I deeply respected him. He's confrontational because he cares. I think the respect was mutual because he called me in this crisis, and we worked together quite a bit when I was a PM.

This almost-customer was stuck on vendor lock-in. Building things on any platform is risky because it's expensive to migrate off of it. The more value a platform provides, the more it costs to get out of it. Mendix is no exception. Our sales people were able to convince most customers that this was not unique to Mendix and that it was alright, but here they didn't budge. Roald knew my reputation for getting sh*t done and that I had done all these crazy projects in the past, like mprtools and Access2Mendix. So, he called me up and said we needed a demo of exporting the logic in a Mendix project to a "normal language like Java". We'd need it Friday. Today was Tuesday and I was flying to a conference that was happening on Wednesday and Thursday. Time was tight to say it mildly.

"What do you need to get it done?"

In times like this, having lots of people doesn't help. A group of 2 to 3 people that already work together extremely well could work. I knew that I needed David. Done. He was on the team. David and I already worked really well together, being good hackers and now both Product Managers. We had a good vibe. He's very clever and had good theoretical knowledge about language fundamentals and transpiling. I don't, but did have all this previous hands-on knowledge.

We knew this prototype was going to be a demonstration that an export to a different language could be done. We were not selling the tool to the prospective customer. We were showcasing that the high-level abstraction in Mendix allowed exports like this. So we were off to the races. Using mprtools we created export-logic for the Domain Model to SQL table creation statements. Not super hard. Then we started extracting the logic in Microflows to Java classes. Ok, this got complicated quick.

In Mendix microflows, every arrow in the diagram is basically a go-to statement. We had to eliminate the go-to's and migrate them to for-loops, if/then/else constructs, etc. . There was some research from the 70s on this topic but it was useless right now. We decided to build the easy cases first and drop down to a "TODO" statement for too complex cases. The people working on the export would have to figure this out for themselves. When I was boarding my plane we had gotten quite far and David worked on it for some more days. At the end there were only very few cases left that the exporter could not handle.

I wasn't there, but apparently the Friday demonstration went great. "If you can do this within a couple of days, it's clear that it's possible and you have a great solution for vendor lock-in." Huge parts of the code were still missing, and the secret ingredients of the Mendix Runtime could not be included in the export at all. How useful was this actually? But, we did a lot better than our competitors and we got the deal! If we were in Pre-Sales, David and I would have probably gotten a fat bonus, but we were in R&D.  Sometimes the intellectual challenge is its own reward. It's certainly a cool story.

2015 - Certinator

My brain is different from that of most people. This is a bit exaggerated, but I either understand something completely or I'm very confused. A coworker recently said that I somehow "know what I don't know". This is a blessing and a curse. A blessing because when I get it I am able to see bugs or solutions in the blink of an eye. A curse because I'm kind of useless and doubt everything until I understand a topic 100%.

Both `git` and SSL/TLS were topics that I didn't get for a long time. When I learned the git commands I was useless, but once I learned the data structures I became a git expert overnight. It's a beautiful idea brilliantly executed. SSL/TLS was the same.

In the Mendix Cloud, customers needed to add custom domains for their apps. As we were enterprisey, this had to be HTTPS but no low-code developers understood how that worked. Once I "got it" I was able to help them out and debug things quickly. I built a small tool "certinator" for this purpose. If you pasted your certificate chain it gave you feedback or autocompleted it. The tool never got far, but building it was a nice journey.
  • Certinator needed to autocomplete intermediate certificates.
  • For that I wanted to find all common intermediate certificates in the world.
  • For that I needed to connect to millions of popular websites and look at their certificate chain.
  • For that I needed to find a list of millions of websites.
  • For that I could use the CommonCrawl dataset and extract URLs.
  • For that I set up a huge EC2 instance in the same region as the CommonCrawl data. I optimized and processed 10TB of data on it within a couple of hours.
  • Then I connected to all domains and collected the certificate chain with a tiny `golang` program.
Watching the machine crunch away the data on 32 cores, and saturating the 10GBps network connection for hours on end was really incredible.

When it was done, I created this blog post: https://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m-domains.html . I posted it with a rather clickbaity title on Hacker News and got to #1 for a couple of hours. My blog post got about 50k hits in one day. That was cool.

We used certinator within Mendix every now and then to debug some certificate issues. My friend Sebastian in the CloudOps team took over that part of the job and I think he used the tool for a couple of years. You can find the source code here: https://github.com/jtwaleson/certinator

A couple of years later I got a message from a startup in San Francisco. They wanted to extract data from the CommonCrawl archives too and had seen my blog post. I was excited and offered to build it for about 3k. It would probably take two days to build, max. They accepted right away, so I was probably too cheap ;) Extracting their data was much more compute intensive, so I went with an ec2 spot-instance fleet. Running the entire archive took a couple of hours with about 100 large machines. It was incredible to see. If you want something like this, call me anytime because I enjoy this kind of work.

2015 - buildpacks for Docker

As I got more familiar with buildpacks and docker, I figured out that we can use buildpacks to create runnable docker images. Not just droplets that you can only run in Cloud Foundry, but plain old docker images. It only took about 100 lines of bash! I still love the elegance of it: https://github.com/jtwaleson/buildpacks-for-docker

2016 - Mendix Cloud v4 - Part 1 - How it got started

We had been building a big new cloud for a year or two, but we did not have a name. I was the Product Manager for "it". Should we call it the Real Mendix Cloud? The Cloudy Mendix Cloud? The Next-Gen Mendix Cloud? Or a brand name like e.g. Heroku, HP Helion, IBM BlueMix were doing. After brainstorming with Roald it dawned on me that this was actually our 4th generation cloud offering. Why not simply call it v4? Customers had always had lots of confusion on where their apps were hosted. They called it "the Mendix Cloud", the "Achiel Cloud" or the "Hans Cloud". If we now said "you are on v2" or v3, and you need to go to v4 because of X, that'd be a pretty simple explanation.

I really liked the v4 name and pushed for it. No one had a better alternative so it was done. In the end the name worked well and I was very proud of my find. So how did we get here? A quick recap.

It had been a long-time ambition of Derek and Johan to become a cloud company. We told customers that they could create their software faster, and maintain it better, but deployment and hosting was still as slow as always. So a couple years before I joined, something happened that started our hosting adventure. We were already running an exchange server, a wordpress website and some other standard IT things when a customer was going live with a big Mendix project. They had a hard deadline because of compliance with a new Dutch insurance law. The only problem was that their IT team could only deliver a new hosting server in about a year. The deadline was end of the month. Not good.

I was not there for any of this, but the story I heard is as follows;

The Mendix guys, creative sales people as always, said "we can do this for you!". They went to their sys admin (Hans) and asked if he could host the app. "Yes we can!". It was a cool engineering challenge, and so customer hosting started. Within a year or two there were all kinds of apps running on virtual machines on colocated hardware. In the beginning there was no budget for anything so the servers were really cheap supermicro machines and the switches were cheap as hell. Disks crashed and at some point the entire SAN was running without any redundancy. Exciting times. We now call this "Mendix Cloud v1".

After some more years there were 3 Ops engineers with about 100 customer applications and they had been more or less "standardized" to a blueprint. The hardware had been expanded, the team was bigger (Frank and Mark joined). The old hardware was phased out and VMs moved to new infrastructure. Things were looking good. But then C-level started talking about Cloud, and that this "hardware thing" was not cloud. Something was brewing.

The requirements for the applications were increasingly "enterprisey". Uptime requirements were very high, VPN access was sometimes mandatory, real-time failovers had to be built. We were building mission-critical stuff. This was "Mendix Cloud v2". Mostly standardized hosting but also some mission critical systems with customizations. There were also still some non-standardized "v1" services running, that slowly had to be migrated to v2.

At this time I joined, as described above in "newnode.py", and Achiel and I started building the "Mendix Cloud". This was when we officially started using that name. Now the main thing about "Mendix Cloud" was to take the hardware blueprint, glue it together in an admin portal (built in Mendix) and let customers manage apps instead of servers. Ironically the Mendix Cloud Portal ran on Cloud v2, which made sense, because we had to be able to start it over SSH if it was down. Of course with Achiel and my limited skillset, we should not host any mission critical stuff yet. So we took on new simple apps, and also started migrating some apps from "v2" to the brand new "Mendix Cloud", retrospectively called "v3". Sometimes it turned out that apps had mission-critical requirements and we had to move them back to v2. Or the other way round. Fun.

We ran the "Mendix Cloud" not on our own hardware, but on cloud providers that were less reliable and much more expensive than the "v2" setup. It was officially "cloud" though because we did not run the servers ourselves, and the costs were OpEx instead of CapEx. We were at a crossroads. Cloud v3 was growing big and expensive, and not capable of serving mission critical stuff. Hans said if we go cloud, we should go really cloud. That basically meant a 12-factor platform and architecture. Now we did not have a 12-factor platform, or even a runtime that could run on it, so we would have to build a lot. That was for now a no-go. Instead we decided to move v3 to our own infrastructure which was more stable, cheap and could support mission-critical apps. We went all in on more expensive hardware (HP ProLiant DL380, NetApp SANs and Cisco switches) and built about 5 full racks worth of equipment in 3 data centers in NL. One by one we moved apps off of Rackspace and Linode and onto our own server environments.

As a medior engineer I had not so much to say about these strategic choices, but by this time we started getting the first Product Managers in the organization. Andrej was the first Product Manager, and after that I immediately applied. Everyone was very much like "wtf I thought you were a nerd that only likes programming". I was told to reconsider and come back in a week. A week later I came back without a changed mind. For some stupid reason I was really determined to do something else than programming. I wanted to have a say in the strategic direction of the product. Johan said he would have to think about it, as Product Managers are typically kind of senior and I was 26 or so. He was 3 years older and CTO since 25 so he always made a point that age did not matter. He managed to convince some people and it worked out, I switched to PM without any additional pay, training or clear responsibilities. Fun times!

I knew Johan and Derek wanted to go to the "real cloud", go global and support massive scale. We had sort of bet on Cloud Foundry already, so we started experimenting with it and see if we could run a Cloud Foundry cluster ourselves. I don't recall that that choice was ever expliticly made, so it probably was made higher up. As PM I was not sure what my responsibilities were, but I assumed my "mission from God" was to get us onto a real cloud using Cloud Foundry. Daniel, Xiwen, Frank and Riccardo started building and playing around with Cloud Foundry and AWS a lot. Once we had some apps we figured out how to build service brokers. Now we knew what we had to build, but it would be a lot.

As PM I was wearing an insane amount of hats, and I loved it. I was doing quite some cloud v4 technical architecture with the team, customer service escalations, joining CSMs on business review calls, managing v1 to v2 to v3 migrations (badly), convincing customers to not build new VPNs, thinking of a pricing strategy for cloud v4, figuring out a staged launch plan and finally creating a migration plan from v3 to v4. All this while v3 was still growing about 50% to 100% year over year and infrastructure was running into new limits all the time. We also needed marketing materials, analyst demos. I had to manage a lot of customer escalations and I did a lot of on-call 24/7 support. I was not supposed to be doing on-call duty, but I had the skills, could use the extra cash and didn't want to burden the team which was already stretched very thin. My to do list was overflowing, I was in over my head, it was pretty stressful and rough, but I'm happy to say that we managed in the end.

In part 2 we look at how we launched Mendix Cloud v4.

2016 - Packing all mxruntimes in git

One of the things that made the buildpack slow, was that it had to download and unpack the whole Mendix Client and Mendix Runtime when building the container image. Over time these got dependencies got bigger and bigger. The images were bloated and needed time to transfer over the network. We needed a way to add a "cache" to our Cloud Foundry clusters, so that the buildpack could just copy the files from disk. We would not even have to include the dependencies in the final image. That would be a lot faster because the apps themselves were usually pretty small.

Storing all the hundreds of runtime versions in the cache of every server was going to take up a lot of disk space though, we needed something more clever.

I realized that most versions were minor updates and that a lot of files were shared between multiple versions of the runtime. I went ahead and created a git repository that stored all of the runtime versions as a tag. You can see the archive here: https://github.com/mendix/runtimes . This repository would be about 1.5 GB and stored 210 runtime versions of +/- 500MB each. Excellent compression!

We would save this repo in our own fork of the cflinux2 base image in Cloud Foundry, so it was already present on all machines in the cluster. Using the copy-on-write mechanisms in Cloud Foundry, we could even do "git fetch" if the app used a brand new Mendix version that was no stored in the cache yet. This was pretty clever. The downside was that it depended on github.com for building apps. Not ideal, but solvable. Another downside was that it would give smart people early notice on when we were releasing a new Mendix version, as they could see the new runtime being pushed.

For some reason we stopped using this approach in early 2018. I don't remember what the team replaced it with.

2016 - The too clever scheduling service

In Mendix Cloud v4 we only wanted to use fault-tolerant services. So no single VMs, but "services" that were "web-scale", "clustered" and "horizontally scalable". Most services we created were request based. So a request would come in, and the service would respond. Simple. We built everything according to the 12-factor app architecture and ran apps on Cloud Foundry. On the AWS side we used lambda and SQS. This was our toolbox for the new architecture.

One service that was not request-based was the backup service. Every night we had to create backups. Users didn't trigger this backup creation, we had to trigger it ourselves. If you use a traditional VM, you'd set up a cron job and you're done. In our new world this was somehow considered bad, but we had no tools in our toolbox to schedule events. AWS Lambda has it now, but back then it did not.

For the backup-service we set up an app in Cloud Foundry that would listen to an SQS queue. If we put a "create backup for app X" job on the queue, the backup service would pick it up and create a backup. This makes sense and is a pretty good architecture. However, we needed something to put these jobs on the queue every night for all apps. Xiwen and I came up with something that was supremely clever. We would put another kind of job on the queue. That job was "run all the nightly backups for <DATE>, and do this at <TIME>."

If the app would see this job, and it was currently past <TIME>, it would trigger the nightly backups AND submit a job for the next day. If it was not yet <TIME>, it would hold the job for about a minute, and then put it back on the queue. The app kept kicking the "can" down the road. Ad infinitum. 

We thought this was clever. If anything ever happened to "the can", there would be no more backups, not this day, nor the next, never. Full of hubris we thought this would never happen. Our team started hiring new people during this time, and Xiwen and I proudly told them about the design. The responses were kind of mixed. When we told Emir he was very impressed by our engineering prowess. When Hans T. came in, he was like "what on earth were you guys smoking".

We went ahead anyway and launched. I remember we had to "insert the magic packet" into the queue by hand to kick-start the process. What happened next was kind of funny. On some nights we had no backups. On some nights we had two, sometimes three. Sometimes half the apps had a backup. We had no idea what was happening.  Hans T had been right, the system was complicated, hard to debug and super fragile.

Because we launched on our free tier we had some time to "screw around", and we got it under control eventually. I don't remember how, but I remember we put duct-tape on duct-tape to make it reasonably reliable. In hindsight we should have been less principled. We had all these super reliable VMs from Cloud v3, and adding a simple cron job to start the backups in Cloud v4 would have been trivial. We didn't do it because we shouldn't "cross the streams". The KISS principle is really important when engineering production systems. "Keep it simple stupid." Better have a small ugly wart than a big one with a lot of make up plastered over it. It was a good lesson for all of us.

2017 - The Reaper

We launched Mendix Cloud v4 by starting the new Free Apps tier on it. These apps without SLA allowed us to take more risks and learn to operate Cloud Foundry. The Mendix Free Apps cluster had enough RAM for about 100 concurrent apps. It could scale up and down, but we wanted to keep the costs reasonable.

An app would run as long as there was HTTP traffic to it. If an app did not have any traffic for 30 minutes, we killed it. If an HTTP request would come in, we'd serve a "loading" page while the app was spinning back up.

We had a small app (The Resumer) that would catch all traffic for apps that were not running & serve the loading page. I believe Xiwen wrote most of it, but my memory is failing me. It would fire a request to the CF API to find and start the app.  The Reaper was its counterpart. It would talk to the CF API, list all running apps. Then call the admin API of the app to see when its last request was handled, and kill it if > 1800 sec. For some reason, The Reaper stopped running once we had more than 15.000 (?) apps in the Free Tier. The cluster started misbehaving badly when out of capacity. We could not figure out why or how, but needed a fix.

When running the same code on my workstation everything worked fine and the cluster went back to normal. So, you can guess what I did. Against all proper engineering practices, I ran "The Reaper" from the workstation under my desk with a cron job for a couple of months. I was a Product Manager and should not be touching code or infrastructure at all, but I couldn't help myself. Later on, Daniel vD (now also a CTO), found that there was a bug in the CF API logic. Without admin credentials it would make a query that scaled O(n2) with the number of apps. Admin credentials didn't need the extra security protection for the query so that's why that was fast. As we were the only idiots in the world running Cloud Foundry with 15k stopped and 100 running apps, no one else had run into this.

Thanks to my friend Daniel you can read the issue here: https://github.com/cloudfoundry/cloud_controller_ng/issues/1272  . When he documented the issue we were apparently at 85k apps. We were growing fast! The Reaper was a pretty cool name for a pretty cool tool.

2018 - Mendix Cloud v4 - Part 2 - How we launched it

TODO

Part 4 - CTO @ easee ('18-'23)

2018 - The easee Wizard Engine

After almost 7 years at Mendix I got a bit restless. My product Mendix Cloud v4 had launched, the big engineering behind it was done and the endless on-call duty was tiring me out. I felt ready for something new. Being addicted to Hacker News I badly wanted to join or start a real startup. A lot was happening. My friend Erik was going to start Soulpicks. Mohsen had some idea with luggage tracking. Rik left for a crazy eye testing startup. Dennis was starting Blockbax with Roy. I talked to some of them about starting something together, but nothing panned out. I had a family and a mortgage, and being the sole breadwinner my personal burn-rate was quite high.

Then I got a message from Rik, who was now the lead unicorn engineer at easee. "We're looking for a CTO. Do you maybe know anyone?" Well, no, but I was interested! A couple of months, sleepless nights and a thorough interview process later, I was the CTO. We were a bit worried about the non-compete with Mendix, but it turned out fine. These things are not really enforceable anyway. This was a hip startup in Amsterdam, exactly what I wanted. Getting in I realized we only had a month or two of runway. The only way to raise some funding was with a CTO in the team, and we needed the funding to pay me. A classic catch-22.

Even before I started full-time I joined for a couple of investor calls, made my first hire (Francesco) and had an epic party after the quarterly kick-off. I'm still banned from dancing on stage at the Chin-Chin club in Amsterdam. This was all new and super exciting to me. Yves, the CEO, was new to startups too, but having an MBA under his belt and a lot of corporate experience was a lot more experienced than I was.

I took 0 days off between Mendix and easee. In hindsight that was not healthy, but we had 3 urgent problems to solve.
  1. Funding.
  2. Scalability. V1 of the eye test would crash with about 10 concurrent users. There was a spike of users after we got on RTL Nieuws and the database did not look pretty after that.
  3. The performance of our cylinder test. A clinical study was underway in the UMC Utrecht, and things looked good, except for the cylinder test. We had a new one but needed to implement it.
We divided and conquered. Yves was naturally on funding (as he has been ever since ;) ). I was on scalability and re-architecture. Rik and Francesco were a couple months more experienced in eye testing and took on the cylinder test. They built this on the new architecture that I was building.

The easee online eye exam is fundamentally a big wizard (plus lots of clever algorithms in the back-end of course). It was implemented as a state machine in the database. Every click went through the state machine in the back-end, a classic MVC in Laravel. This was slow and did not scale because the database architecture was a mess. With a client-server architecture we could never get to sub 100ms responsiveness. I decided to go for a Single Page App built with Vue, with most of the logic of the wizard in the front-end. I had no experience with Vue, but loved its pragmatism. Now I had to figure out how to create a simple engine for displaying the screens in the correct order.

We needed to show screens one after another, and depending on the input of the user, go through a different path in the wizard. We could simply point the "next" button to the next screen. But some screens were re-used, so that was a no-go or we'd get a loop. I felt like the code would get messy, but there had to be a clean solution. It took weeks of banging my head against the wall, but then it clicked. I learned how async/await worked (I never understood it before). I realized the logic of a wizard is just a classic program. Each screen has input (the props) and output (whatever the user selects). If we wrap this in a promise we can "await" the user's action. Instead of "awaiting" a database or remote server, we await the action of the user. I thought this was extremely clever.

So that's what I built. A small engine of about 100 lines of code that can show a screen, await the user's action on it and return the output to the main program logic. It can serialize the data and store it in session storage so that you can safely reload the browser. The serialization also allows the state to be sent to another device and continue there. I can't share much more details, but it's really cool. 

This simple engine allowed the developers in the team to just work on the screens and keep the codebase clean. It also made the flow of the eye exam easy to understand, as it's just a program with functions, output and variables.

We quickly rebuilt the exact same product as v1 on the new engine. Then we added the new cylinder test and launched the second part of the clinical study with it. Yves managed to raise a good seed round with Nimbus Ventures. All of this within 4 months of me joining. In November 2018 we put v1.5 of the product live with the new engine. Since then it has powered hundreds of thousands of eye tests and had almost no bugs. I am very proud of the easee wizard engine.

2019 - The easee Diff Viewer

Medical Devices need an IFU document. The "Instructions For Use". It's called the User Manual in the normal world. The consumerisation and digitalization of healthcare blurs the contrast between the Medical Device world and the normal world. When is the last time you looked at the user manual for an app on your phone? Do most apps even have one? Probably not. The User Interface should be be self-explanatory. Why would it be different for a healthcare app?

Having an IFU is kind of a ridiculous requirement if no one will ever look at it. As far as I know, even the regulators won't take more than a cursory glance for low-risk healthcare solutions.

With easee we decided that the UI is the IFU, and for purely regulatory reasons we needed to create a PDF document out of it to show to the regulators. But how? Doing this by hand would be error prone and a lot of work, as we have a new version every couple of weeks. Ideally we would automate the PDF generation for every release. Now this stupid regulatory requirement had turned into an interesting engineering challenge! I had the feeling that one of the puzzle pieces was hiding in the wizard engine, and also that this PDF generation would only be the beginning of something beautiful.

The wizard engine can render pages based on just page parameters. If we made a "debug" renderer, we could show individual pages if we knew the parameters. Using selenium we could then create a screenshot of the page. But how do we capture all the pages in the eye exam?

Well, the wizard engine can also capture all the page inputs and outputs in SessionStorage. We can store this "execution tree" scenario in a json file. So how this works is: you manually execute an eye exam, and at the last step press "store this scenario". Now we have captured all the page inputs. We store this json file next to the source code, and using a small debugging tool can jump to any step in the exam to debug things.

In CI we can load these captured scenarios, and one by one render the pages in it and capture screenshots. If we create a small HTML page with all these screenshots in it, we should be good! We now have a simple IFU document of just screenshots. But we also would like some accompanying text. In Vue you have "single file components" that specify template, script and style tags. We added some other tags like "documentation". This is parsed in CI and added to the html page, and now we had auto generated screenshots per release! IFU done!

Then I realized that we would also like to make really really sure that we don't unintentionally change anything in the UI of the eye exam. It would be neat if we can compare pages from before and after, and before a release check for "visual regression" problems.

The diff viewer could do this too! During CI we were already storing the component descriptions and screenshots to AWS S3. We needed an app to quickly compare all screens in two versions of the software. Due to the combination of screens x screen sizes x page parameters, we had close to 5000 different screenshots. How to do this efficiently? I learned about image diffing and perception hashes. In the CI run I added a perception hash to the metadata files in S3.

I created a quick & dirty Vue app to load two different versions of the exams and show the differences in screenshots. If there was a diff in the perception hash, it sends two images to a lambda function that shows the differences highlighted.

As CTO you have to say that it was a "team effort", and that's true for almost all of the things that we do. I don't code much anymore. But while the team was working on the important roadmap stuff I got inspired. I built the entire CI screenshots scripts and the diff viewer in about two weeks by myself. I'm not really proud of that, I shouldn't be taking the fun projects for myself. Later on Rik and Mason started adding some features, and they took part of the ownership. Thanks for working on my hacky code guys.

The easee diff viewer is a pretty cool multi-functional tool that compares thousands of images within seconds. It's awesome. We use it to verify that things looks good on multiple browsers and that we don't unintentionally change anything. I think it could be a separate SaaS product by itself. If only we had infinite time to launch new products.

2019 - easee RemoteControl

The easee wizard engine was quite cool, and we used it to structure the code for our online eye exam. It executes whatever you see on your computer screen. It was not all good though. At some point in the exam your smartphone is paired using a websocket. The code for sending the phone what to display was a bit ugly. I had an inkling that there was a cleaner solution.

We ended up with a special Vue component called "RemoteControl". This component has three props:
- remoteComponent
- remoteProps
- callback

When you use the RemoteControl component, a couple of store methods are called. These will serialize the remoteComponent, remoteProps and the callback and send them over the websocket. At the other side the specified component is displayed with the props. A special prop for the callback function is created too, so the user input from the remote control can be sent back to the main computer.

As the remoteProps and remoteComponent are reactive, the phone screen updates whenever the main computer decides to update the remote. It's magic.

I can't share the code here as it's a bit entangled into our codebase, but the idea is so simple that you can probably create it within a day. There's a couple of caveats:
- the remote checks if it can find the remote component, if not, throw an error
- there should be only one RemoteControl active at the same time
- as network messages can be delayed, add an envelope with a random ID to see if the answer is actually sent to the current instance of the RemoteControl. If not, it might be from a previous version and you should discard it.

As with the wizard engine, having the right abstraction layer can make the codebase a joy to work with. It certainly does in this case!

2020 - Translation Mixer

TODO

2020 - easee TeleVisus

Early 2020 was a strange time. I remember seeing some videos of sick people in China around January 1st, but my memory might be off. By late January it was clear we were in for trouble. In February it got closer and closer. One coworker next to me actually got covid, and I might have been infected too but did not have a lot of symptoms, I still don't know. At the end of February we went for a pubquiz with easee, and I told my coworkers "In two weeks time this won't be possible anymore". Indeed. Two weeks later to the day, all non-essential businesses had to close their doors. As a medical device company we technically were an essential business, but we decided not to play that card. Some other software companies in our sector did, just because they wanted butts in seats. Bastards.

We had just had our first audit in November 2019 and now had a TUV Rheinland certified Quality Management System. This is a whole set of procedures that you have to follow when you do your day-to-day work, to guarantee safe and effective products. Setting it up had cost me blood, sweat and tears and my hair was much more gray than before. Our product certificate for our measuring function was not finished though. During Q1 2020 we had difficult calls with a tough German auditor. Francesco and I hated those as we were nice and non-confrontational. Robert loved it. Yves was somewhere in the middle but was extremely driven to get our certificate. I'd rather give up. I took a step back during those calls. Live and let live I suppose. Early March the auditors told us that our product was good to go, it just needed some paperwork on their side. It would take another 4 weeks. Great!

On the evening of Thursday March 12th the government announced that offices could not open the next day. We closed the office and everyone had to work from home. A disorienting change for most of our employees. For me the spring of 2020 was great, I could spend more time with my kids and didn't have to commute for a change. Alas, there was no rest for the wicked. That same day, Yves had an idea to offer a limited test for hospitals as soon as possible. As care had to be offered remotely, hospitals needed to check patient's eye sight remotely too. This was a great opportunity for us and we all wanted to do our part. I asked for volunteers to work in the weekend. Everyone raised their hand. We jokingly said "What else are we going to do?" but actually we wanted to make a difference and do something meaningful. That weekend we made a plan and the team started running. We tried to do things according to our brand new QMS as much as possible, but our enthusiasm sometimes got in the way.

On Tuesday we had the management team offsite to prepare for Q2 2020. This was Dennis (Head of Business Operations), Yves (CEO) and myself (CTO). We had already booked an Airbnb and if we interpreted the government rules a bit liberally we could have the meeting in person. I felt slightly guilty about this, but it was good as we were all a bit down. The stock market was crashing, travel was suspended, people were sick and dying. We could use the company. All of a sudden the mood changed. Yves now fully realized what a giant opportunity we had in front of us. The world had to adapt itself to remote work, remote care, remote everything! And what did we just get approved as a medical device for the entire EU? A fully remote online eye exam!

This was the most chaotic management team offsite I've ever experienced. Yves' optimism always makes him a bit over-enthusiastic and hyperactive, but here he went up to 11. And rightly so, things were happening! We had one hour of useful meeting time in the twenty hours we had scheduled. The rest of the time Dennis and Yves were either calling our business partners, prospecting or preparing press releases. I was on Slack with my team constantly. Mason, Rik, Francesco and Danielle were going full-speed ahead with the new test, and I tried to bring some structure and answer a lot of questions. We had amazing people in the team and I was very proud of having hired them. Iris and some other people were updating the website left and right. We called the new test TeleVA for Tele Visual Acuity. Yves made an executive decision and renamed it to TeleVisus. I really liked that name, but we now call it the "easee online visual acuity test".

We finished everything within two weeks, including usability testing, verification and validation. The TeleVisus test was put in our product in v1.6.118 and released on March 31st 2020. In any normal development cycle, it would have taken at least 6 months. This insane inefficiency is the sad reality in most companies, most of the time. This might be a good time to quote Colonel Kurtz from Apocalypse Now: "If they were committed, this war could be won with a fourth of our present force.". The big question for all managers is how to consistently get to this level of commitment at least some of the time. I don't have the answer. If you do, call me.

Later that year we got in trouble for the TeleVisus release. We launched it without a measuring function claim, but when we finally got our certificate for that, we also applied it to TeleVisus. The auditors were not amused, as they had not reviewed this new part of the product. We should have explained this to them, but I was actually not aware that we had to as we did not yet have a clear "significant change" procedure.  We got a couple of minor nonconformities for this, but in the end we were good. The measuring function was identical, and everyone understood we were trying to help hospitals during a huge crisis.

TeleVisus is clearly my #1 project when it comes to teamwork. Technically it was not super challenging because we had all the components already. But the way we rallied together in a crisis situation.... it was simply incredible. A special memory full of light in those dark times.

2020 - QMS Integrity Checker

When we set out to get external certification for our product, we were in a bit of a bind. There's a whole bunch of international standards that have to be followed, and we had almost no experience with these. For Software Medical Devices there's at least ISO 13485, ISO 14971 and IEC 62304. The ISO 13485 is interesting, as it's a set up of requirements for forming a "Quality Management System" (QMS).

Why is it interesting? Well, there's many ways of achieving high quality products, and I believe strongly in "move fast and break things". That's a good approach in general and many companies are successful with it (Facebook, SpaceX). It takes guts though. The failures will happen, and you need people with a high risk-tolerance to accept them. One of the reasons SpaceX can pull it off, but NASA can't, is that the tax-payers in the US would probably not accept many failures, and Elon does. It kind of makes sense to say that pacemakers should not be built this way, but then again, you could say the same about rockets.

It's safe to say that most medical devices don't take a "move fast and break things" approach. Rather the opposite. They try to prevent failure by over-engineering up-front as much as possible. For most people from the software world this is a bit of a shock. For me it certainly was. I had no idea what a QMS was, but we did have to build one.

You can not read the rules for a QMS for free, you have to buy the document online for a hefty fee. When we found the standard for relatively cheap at evs.ee we could finally read how to set up a QMS. Or so we thought. It's ironic that the 13485 standard exists to make you explain your processes and records in writing, but the standard itself does not contain any explanations. It just says "do X, do Y, do Z". Never "Rule X exists because manufacturers often have unclear descriptions of responsibilities within the organization, so it's unclear who can sign off on certain decisions". No, it just says "The organization shall document roles and responsibilities." It's a bit like reading a compiled program, stripped of all comments or useful variable names. This gives you a lot of freedom for interpretation, but no clear path forward.

Once we had more or less deciphered what we had to do, we figured out that a QMS is a bit like a computer program. There are users, roles, processes with inputs and outputs. It's super systematic. With our limited understanding we set out to build a computer program to contain our procedures, technical documentation and records. It would publish everything as a static website with a git-based database. This was fun and pretty cool to build, but we kind of got lost in the engineering details instead of the QMS details. We also realized that we'd need engineers to operate every part of our QMS because it was using git. That was not so good, because it excluded a large part of our company from contributing. We could also not download all html pages and submit them as PDFs to an auditor. This was very bad. A couple of weeks before we had to submit our QMS for external certification, we reverted back to the industry standard approach. That is A4 documents in Google Docs, then printed out and put signatures on them. Terrible, but it got the job done. We got certified with this system in 2019. We abandoned the QMS static page program we had built and loved.

The systematic approach was still there though, hidden within our Google Docs and Sheets. In the little spare time that I had I wanted to see if I could use Google Drive, Docs, and Sheets APIs to make a sort of "integrity checker" for our QMS. I had a lot of fun to talk to the APIs and parse the document structure in a Google Doc. At the end I even had a Google Docs to Markdown converter. What's more, I was able to parse all the roles, responsibilities, cross-links to other documents and much more. E.g. if you refer to a "Product Owner" in a procedure, but you haven't defined that role within your organization, it will throw an error. It scans hundreds of documents in a couple of minutes. Although it's still in its infancy, I run the integrity checker every now and then and point my colleagues to tiny mistakes in their documents. That's pretty fun and I look really smart. In theory this program combines best of both worlds: your regulatory people deal with Google Docs but the QMS can still be completely checked automatically using software. Might be a cool business idea actually. The plan was to incorporate this in some kind of CI/CD or monitoring system, but I never prioritize it and it just runs on my laptop. It's written in Python (what else) and the code is ugly but I love it.

I personally struggled a lot while getting certified, it was probably the most stressful period of my life. Afterwards I found out that we are not alone, many other software companies are struggling with the same things, and we could really help each other out. I got in touch with Oliver (hi!) and he set up OpenRegulatory.com with it's Slack group and I started a bi-monthly virtual SaMD meetup. Both of these communities are thriving! Now, if you have any questions, just ask on Slack and experts will point you in the right direction right away. I dearly wish I had access to this in 2019. It's not a software solution, but I'm extremely proud of starting the community.

Now that I'm a bit further in the regulatory field, I've also discovered that the rules do allow for a bit of "move fast and break things". The regulators realized that not everything can be checked beforehand, and rely on post-market surveillance to compensate. This catches problems that were not detected before. The feedback, CAPA and vigilance processes as described in ISO 13485 force you to learn from mistakes, and explain the serious problems to authorities. Realizing this made me feel very wise. Like a grown-up software rebel that realizes the entire world was thinking the same thing all along. Here's a quick shout-out to the people thinking about Chesterton right now.

2021 - The WaterRower Camera

I was getting more and more heavy during this period, almost reaching 90kg / 200lbs. I never exercised but I had bought a rowing machine with a water tank because I wanted to exercise, in principle. The machine was bought in 2019 just before the pandemic and during the boring early pandemic months it was a welcome way to distract myself. For some weeks I started rowing like a maniac but of course it wasn't sustainable. The machine started collecting dust.

It had a small "workout computer" displaying the stats, but it was very dumb and not connected to anything. For some reason I thought it would be a good idea to measure my stroke speed, depth, and power more closely. The (black) elastic band is wound up around a cylinder. By painting the cylinder white and tracking with a camera how much black/white pixels are visible, it shows how far you've extracted the cord. Doing that a lot of times per second you can see stroke depth, speed, power, and recovery speed. I attempted to do this with a cheap ESP32 board with a camera but found that I could only get 1 FPS. I hooked up a raspberry pi zero 2 w and reached 60FPS. Then I built a simple web interface which I used to exercise about 5 extra times.

Then my wife's cousin Johannes came by, saw me rowing and laughed his a** off. Apparently my rhythm was completely wrong and I should do short and very fast pulls, then a slow recovery. I had taught myself rowing in the wrong way and now couldn't get used to the right away anymore. I haven't rowed since, but I discovered that running is very nice. In the summer of 2022 I lost 13 kgs and got down to 77kg and feel a lot healthier.

2021 - FDA Crawler

Some industries see a lot of competition even though there's not a lot of commercial traction yet. There the battles are fought not over customers and revenue, but over patents and certifications. With easee we were operating in a similar environment. It was very important for me as CTO / product person to figure out what our competitors were doing. One of the tools that I developed for this is the FDA crawler. It scrapes the FDA API for product and company registrations and serializes it in a git repository. Every week the script runs and with the git history you can see exactly if there are any created, updated or deleted listings. Any new company or product category is instantly visible. Well, sort of, because the database is only updated manually once every couple of weeks. We typically see news via a google alert weeks before it is visible in the official channels. Nevertheless, this serialization to git is a nice pattern that I hope to use for more purposes. It helps us keep track of any new competitors in the USA. It's a tiny script but really effective and sure beats checking the database manually.

2022 - Cowboy e-Bike Remote Control

In early 2021 I was a bit down. It was winter, some team members had just quit and I needed to distract myself. Due to covid lockdowns and boredom I gained more weight and drank more. I needed exercise (yes I had a water rower but I was too lazy to use it. I had to cycle every day so a bicycle would be a good idea).

I started looking at e-bikes, because having experienced the rentals on a trip to Berlin I was sold. It makes cycling even more fun, and you end up taking the bike for most of your trips.

The two options were the VanMoof S3 and the Cowboy V3. The Cowboy won. Less quality issues, sleeker look & minimalistic approach. I love this bike. It brings a smile to my face whenever I step on it.

Sometimes the smile turned upside down because my phone would not connect to bluetooth though. Stephanie also uses the bike and her phone would be connected instead, or something else went wrong. It was just not super reliable. If things that are almost magical don't work reliably, it's very annoying for some reason. You know how great it can be, so why can't it be that great all the time? If you already read about InstaDeploy you can see that I'm guilty of this as well in my own creations.

Anyway, I learned about the "hacking" apps for the Cowboy bikes, and because some were open source, I saw they used bluetooth. I then started a 1 year quest to create a bluetooth-powered car key for my e-bike.

I had already dabbled a bit with ESP8266 and ESP32, but only using some simple wifi apps using Arduino IDE. Here I was out of my comfort zone.

It's a small-sized hardware solution, no prototyping boards allowed. I had never done that before. The key should work from a coin cell for 1+ year and optimized for ultra low power. Not done before. It also needed to work with Bluetooth, BLE, whatever. No experience whatsoever. Fun, let's go!

I quickly saw the the ESP32 would not work, it used too much power for a coin cell and was quite bulky. So, the nRF52 series might be an option.

I had to learn a lot. Ordered a nRF52-DK (Development Kit) and started creating some code for connecting to BLE devices. As the Nordics team is transitioning between two versions of their SDK, I was very confused most of the time. The documentation seemed to apply and then not apply to my situation at random.

Anyway, I persevered. I did it in evenings and weekends, and I believe it took me about 2 or 3 full time weeks of work. You can view the end result here: https://blog.waleson.com/2022/03/the-worlds-first-cowboy-key-fob.html

I'm still working on getting the version with the PCB to work. For some reason the power cuts out after an hour or two, even though there should be no current. Still debugging that one.

I'm currently using my prototype. It's beautiful on the outside but ugly inside. Every day I walk to my bike, press the button on my key and unlock it. Love it.

Comments

  1. a great read, an insightful as well ;)

    ReplyDelete
  2. This is an excellent read, thank you for sharing your journey :)

    ReplyDelete
  3. Man, thanks for the SnelTrein app, there was a time when it was the only way to see the schedules.

    ReplyDelete

Post a Comment

Popular posts from this blog

The unreasonable effectiveness of i3, or: ten years of a boring desktop environment

Idea time: RFID+E-Ink, electronic price tags without batteries

Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS