[{"data":1,"prerenderedAt":54},["ShallowReactive",2],{"$f3att4jbeme16g":3,"recommended":23},{"title":4,"path":5,"date":6,"updated":7,"tags":8,"excerpt":12,"readingMinutes":13,"commentCount":14,"html":15,"comments":16,"next":17,"prev":20},"[TSTIL] Certinator","\u002F2022\u002F10\u002Fcertinator\u002F","2022-10-05T07:05:00.010Z","2024-06-11T08:17:33.258Z",[9,10,11],"mendix","the-software-that-i-love","infrastructure","[ This is a part of \" The Software That I Love \", a series of posts about Software that I created or had a small part in ] My brain is different from that of most people. This is a bit exaggerated, but I either…",3,0,"\u003Cp>[ This is a part of "\u003Ca href=\"\u002F2022\u002F10\u002Fthe-software-that-i-love\u002F\">The Software That I Love\u003C\u002Fa>", a series of posts about Software that I created or had a small part in ]\u003C\u002Fp>\n\u003Ch2>2015 - Certinator\u003C\u002Fh2>\n\u003Cp>My brain is different from that of most people. This is a bit exaggerated, but I either understand something completely or I'm very confused. A coworker recently said that I somehow "know what I don't know". This is a blessing and a curse. A blessing because when I get it I am able to see bugs or solutions in the blink of an eye. A curse because I'm kind of useless and doubt everything until I understand a topic 100%.\u003C\u002Fp>\n\u003Cp>Both `git` and SSL\u002FTLS were topics that I didn't get for a long time. When I learned the git commands I was useless, but once I learned the data structures I became a git expert overnight. It's a beautiful idea brilliantly executed. SSL\u002FTLS was the same.\u003C\u002Fp>\n\u003Cp>In the Mendix Cloud, customers needed to add custom domains for their apps. As we were enterprisey, this had to be HTTPS but no low-code developers understood how that worked. Once I "got it" I was able to help them out and debug things quickly. I built a small tool "certinator" for this purpose. If you pasted your certificate chain it gave you feedback or autocompleted it. The tool never got far, but building it was a nice journey.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Certinator needed to autocomplete intermediate certificates.\u003C\u002Fli>\n\u003Cli>For that I wanted to find all common intermediate certificates in the world.\u003C\u002Fli>\n\u003Cli>For that I needed to connect to millions of popular websites and look at their certificate chain.\u003C\u002Fli>\n\u003Cli>For that I needed to find a list of millions of websites.\u003C\u002Fli>\n\u003Cli>For that I could use the CommonCrawl dataset and extract URLs.\u003C\u002Fli>\n\u003Cli>For that I set up a huge EC2 instance in the same region as the CommonCrawl data. I optimized and processed 10TB of data on it within a couple of hours.\u003C\u002Fli>\n\u003Cli>Then I connected to all domains and collected the certificate chain with a tiny `golang` program.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Watching the machine crunch away the data on 32 cores, and saturating the 10GBps network connection for hours on end was really incredible.\u003C\u002Fp>\n\u003Cp>When it was done, I created this blog post: \u003Ca href=\"\u002F2016\u002F01\u002Fparsing-10tb-of-metadata-26m-domains\u002F\">https:\u002F\u002Fblog.waleson.com\u002F2016\u002F01\u002Fparsing-10tb-of-metadata-26m-domains.html\u003C\u002Fa> . I posted it with a rather clickbaity title on Hacker News and got to #1 for a couple of hours. My blog post got about 50k hits in one day. That was cool.\u003C\u002Fp>\n\u003Cp>We used certinator within Mendix every now and then to debug some certificate issues. My friend Sebastian in the CloudOps team took over that part of the job and I think he used the tool for a couple of years. You can find the source code here: \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fjtwaleson\u002Fcertinator\">https:\u002F\u002Fgithub.com\u002Fjtwaleson\u002Fcertinator\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>A couple of years later I got a message from a startup in San Francisco. They wanted to extract data from the CommonCrawl archives too and had seen my blog post. I was excited and offered to build it for about 3k. It would probably take two days to build, max. They accepted right away, so I was probably too cheap ;) Extracting their data was much more compute intensive, so I went with an ec2 spot-instance fleet. Running the entire archive took a couple of hours with about 100 large machines. It was incredible to see. If you want something like this, call me anytime because I enjoy this kind of work.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>next\u003C\u002Fstrong>: \u003Ca href=\"\u002F2022\u002F11\u002Fbuildpacks-for-docker\u002F\">2015 - buildpacks for Docker\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>\u003Cstrong>previous\u003C\u002Fstrong>: \u003Ca href=\"\u002F2022\u002F10\u002Finstadeploy\u002F\">2015 - InstaDeploy\u003C\u002Fa>\u003C\u002Fp>\n",[],{"title":18,"path":19},"[TSTIL] Mendix2Java","\u002F2022\u002F10\u002Fmendix2java\u002F",{"title":21,"path":22},"[TSTIL] mxplient","\u002F2022\u002F10\u002Fmxplient\u002F",[24,29,34,37,41,44,47,51],{"title":25,"path":26,"note":27,"series":28},"Mapping out Dark Code","\u002F2026\u002F03\u002Fmapping-out-dark-code\u002F","2026",false,{"title":30,"path":31,"note":32,"series":33},"The software that I love","\u002F2022\u002F10\u002Fthe-software-that-i-love\u002F","Series · 36 posts",true,{"title":35,"path":36,"note":27,"series":28},"Navigating the Fog of War: Age of Empires, AI Agents, and the Rise of Dark Code","\u002F2026\u002F05\u002Fnavigating-fog-of-war-age-of-empires-ai\u002F",{"title":38,"path":39,"note":40,"series":28},"The Future of Programming Systems - four thoughts","\u002F2024\u002F09\u002Fthe-future-of-programming-systems-four\u002F","2024",{"title":42,"path":43,"note":40,"series":28},"Always Optimize the Feedback Loop","\u002F2024\u002F07\u002Falways-optimize-feedback-loop\u002F",{"title":45,"path":46,"note":40,"series":28},"\"Security Is Our Top Priority\" is BS","\u002F2024\u002F07\u002Fsecurity-is-our-top-priority-is-bs\u002F",{"title":48,"path":49,"note":50,"series":28},"10x teams","\u002F2025\u002F02\u002F10x-teams\u002F","2025",{"title":52,"path":53,"note":50,"series":28},"Hiring, from the Other Side","\u002F2025\u002F10\u002Fhiring-from-other-side\u002F",1792876808935]