Three Thoughts on Dark Code

First published on

A couple of weeks ago Dan Shapiro's blog post "The 5 Levels" (about Agentic Engineering) did the rounds. He coined Level 5 the "Dark Software Factory". So what comes out of those factories? I'm calling it Dark Code.

Dark Code: lines of software that no human has written, read or even reviewed.

Two weeks ago I presented a talk "Mapping out Dark Code" at the OpenAI Codex meetup in Amsterdam, and now people start conversations with me and mention Dark Code. So before this goes out of control, let me write some thoughts.

Tldr:

  1. Will my codebase become an unholy mess if no one looks at the code?: Yes, this is an unsolved problem.
  2. Will there be new programming languages if humans don't care about code anymore?: Doubtful, and that's a shame!
  3. Dark Code is already here: In enterprises no one knows the code anyways.





1 - Will my codebase become an unholy mess if no one looks at the code?

With how agents work nowadays, I see parallels with evolution chipping away at our ACGT base pairs. It's just trying some things until it works. Likewise, in software, a prompt will just find the easiest way to achieve its goal.

If the system is big enough, a random request in a prompt will not trigger a full refactoring or a new architecture. I feel architecture will get stuck in local maximums quite quickly.

Yes, without new tooling for architecture, backpressure on the agents when complexity goes too high, your codebase will become an unholy mess. However, there is a lot of development in this field now, so some things will emerge.

2 - Will there be new programming languages if humans don't care about code anymore?

The SWE Bench performance on new or unknown languages by LLMs is abysmal. I saw an article on this recently but can't find it anymore. There's no or little training data on these new languages, so the LLMs have to use their reasoning and can't rely on "muscle memory", to put it in lay-men's terms.

What does this mean for programming language research?

A) All coding agents are generating ever more code in current, general purpose, languages, and it's a bit of a vicious cycle. The more code is generated, the more models will be trained on that. It's hard to get good model performance on a new language, and thus it's harder to generate enough code to be trained on.

B) Previously humans cared about their code. When they saw a way to create a better language, some people were crazy enough to create a new one! Because they cared so much about the software systems they built. No wonder, they were living in their codebases all day. Now that we let most code be generated by our agents, do we still care enough to create a new language? I don't think so.

But, as I've written in the future of programming systems I believe there's a LOT of room left for better programming languages. I believe the Low-Code movement was partially about this. Standardizing what we were trying to build.

A language / linter / syntax should know exactly where your application talks to the outside world. Where there is Personally Identifiable Information flowing through your systems. Instead, all major current languages are general purpose languages, and unconstrained. Unconstrained sounds good, but actually, constraints allow you to speed up tremendously.

Will we ever get nice things, or are we casting the current status quo in stone. Will Rust be the best language forever?

3 - Dark Code is already here

Dark Code is not that binary. Code I've written 1 year ago is already a lot darker than the code I'm writing today. Enterprises are full of code written by teams that are no longer there. That code is pretty dark.

My startup is trying to shed a light on large software systems that are too complex to understand, unless you take a birds eye view. I'm a bit scared that it might not be relevant in the world of Dark Code, because people don't seem to care about code anymore. On the other hand, there is now more code than ever, so I'm betting that people will want to work on this higher level. Maybe we're building the IDE of the future?




Thanks for reading. If you liked this post, you might also enjoy my thoughts on the future of programming systems.


Discuss on HN, comments on the blog are mostly spam.

Comments

Post a Comment

Popular posts from this blog

The long long tail of AI applications

First published on
Image
Last year multiple companies asked me for advice. " We are evaluating this AI powered product, but do you think it makes sense at all? It seems a bit niche and we think ChatGPT might make this entire thing obsolete soon. " Long tail - credit to Wikipedia My answer so far has always been: "No, bare LLMs are not going to compete with this product." I think that people are failing to understand the distinction between different classes of AI companies. I see it like this: Foundational AI - creating models like GPT-4 (text), Sora (video) etc. Applied AI - using existing models to create smart applications. Note that there are also companies not focusing on AI - they will start lagging behind, let's forget about them ;) There are orders of magnitude  (!)   more companies that will deal with Applied AI than Foundational AI, and they will be very busy for decades to come. Here's why: To get the most out of LLMs, you have to ask the right questions. LLMs have acc...
Read more

"Security Is Our Top Priority" is BS

First published on
A couple of years ago I was asked to give a conference talk about software security. Well, actually I wasn't really asked, my company bought a sponsorship package with a speaker slot and I replied to the internal email asking for volunteers  🤣  Anyway, while preparing my talk, I realized a couple of important points about security that have not left my mind since: Security is limitless . You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc. The needs of security are opposed to the needs of a convenient user experience . Improving one typically hurts the other. Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that...
Read more

Bakelite to the Future - 1950s rotary phone ESP32 bluetooth headset

First published on
Image
Edit: since creating this post I've also created a video about this project on YouTube. See it below or jump straight into all the details below. Here's a very beautiful antique rotary phone. It's a Dutch PTT phone model called simply "1950" . But, what's this at the back? A USB-C port? A peek inside shows us some weird things! How did that LiPO battery get there? Ok let's quit the jokey intro, this was all me. I've had this phone for about 15 years and over the last couple of weeks I've converted into a fully functioning Bluetooth headset using an ESP32 microcontroller. In this post: - 1 - Features - 2 - Lessons Learned - 3 - Pictures and story!  of how it got made (scroll down!) but first... Does it work? This morning was the first Google meet I've used it in: A participant reported on the sound quality "I didn't know how much I missed the different sound quality! Beautiful. We could hear you very well, not distorted. I wish I record...
Read more