Always Optimize the Feedback Loop

First published on

Speed, I am Speed. -- Lightning McQueen

Some thoughts about the impact of feedback loops in startups and software.

Parts:

  • Intro
  • Framework
  • Observations
  • Take-away

Intro

Feedback delay is the time between doing something and seeing whether it was successful.

If the feedback is slow, we have a "lagging indicator". It becomes hard to link causes and effects, and in consequence it becomes hard to optimize the performance. Fast feedback makes things easy.

How quickly the feedback arrives is critical. If it takes milliseconds to seconds, it works at the speed of the brain and enables a state of flow. If it takes decades to centuries, it needs a multi-generational organization like a government or organized religion to run it. Somewhere in the middle is the realm of persons, teams and companies.


A framework of feedback time

Here's a quick table of feedback times, in three groups, with each roughly an order of magnitude difference.


    C is the level where (large) organizations can run experiments without seeing the results for a long time
    • C4 centuries
    • C3 decades
    • C2 years
    • C1 months
    At B a person or team can have the patience to execute this

      • B4 weeks
      • B3 days
      • B2 hours
      • B1 minutes
      A is the speed of the brain, the feedback arrives so fast that it feels effortless.
      • A4 seconds
      • A3 100+ milliseconds
      • A2 10+ milliseconds
      • A1 1+ milliseconds

      Observations

      With this mini framework in mind, here are a couple of observations.


      1 - Startups

      Startups are viable if they do something 10x better. Each of the steps is a 10x improvement, so if you manage to create a solution that brings feedback time down one step on the ladder, you have a viable startup. At least according to Peter Thiel.

      2 - Faster is always better

      Faster feedback is always better and never worse, but there are diminishing returns. E.g. when I'm typing and it takes 4 milliseconds to render to the screen, it's already imperceptible, and improving it more offers no benefit.

      When looking to improve something, check what level it is at right now and what would count as imperceptible. The difference is the space that a new product or service could enter.

      For example, in a regulated industry I will get feedback from an auditor at C1 (a couple of months) but my life would be so much better if they replied at A4 (seconds) and I can have a dialogue. So: there's an enormous opportunity here for a regulatory body that promises responses in B1/B2 (see e.g. Scarlet) or for a group of experts you can chat to instantly (OpenRegulatory Slack).


      3 - Multiple orders-of-magnitude improvements shake the world

      Improving by 1 step is great, but something weird happens when we take 2 steps down the ladder or more. Things that were previously impracticable or downright impossible, now become in reach.

      When jetliners were invented, we could travel around the world in B3 (days) where in 1930 it still took my grandfather C1 (months) to go from Indonesia to The Netherlands. Jet travel has certainly changed international business, politics and tourism.

      A recent example are LLMs. Getting a book summarized would take a human B2 (hours), and ChatGPT does it in A4 (seconds). By making it two orders of magnitude faster, we just went into flow-state territory!

      When a new technology makes multiple orders of magnitude improvements, the world needs a couple of decades to adjust to this new reality.

      4 - Software vs Hardware

      Most software development features are in B, while hardware is in the lower Cs. They are wildly different industries because of this, with different approaches to QA.

      Software developers that go into regulated industries see their feedback time going from B1 (when automated tests tell me there is a problem) and B3 (when customers tell me if they like something) to C2 (the regulators will review your documentation and give you feedback in about a year). Talk about culture shock!


      5 - Hunt for the feedback loop

      Whenever I am stuck on a problem, I have bad tendency to get blocked and not knowing where to start. When I notice this happening, I've learned to try to find something to do that works towards the goal and that gives me feedback the fastest.

      In programming there are feedback loops that will always get me into a hyper productive state:
      • Getting compiler / linter errors in my IDE. When I've refactored code or when I introduce a linter to help clean things up, I get a bunch of new errors that I just have to fix one by one.

      • I execute the program and get an error, then I fix it and go to the next error.

      • Getting CI test results (e.g. speed of the program, memory usage, code coverage). Changing something and seeing the result of my change gets me feedback pretty quickly, but CI is often in B1/B2 (minutes/hours) and unfortunately frequently gets me out of flow state.

      • Create a to-do list of things to finish, and try to get it to 0 as quickly as possible.


      SpaceX famously achieves rapid progress through rapid feedback cycles compared to the upfront design work at e.g. NASA.

      Their long-term goal is to colonize Mars, which is clearly in C4 / C3 (centuries / decades). They won't know if this will work until it's done. Those kind of grandiose projects have a bad habit of not getting started. SpaceX brilliantly broke the problem up into multiple milestones, and got a feedback loop going towards milestones 1 - launching Falcon 9, 2 - making the boosters re-usable, 3 - building a bigger rocket which is entirely re-usable.

      Their results are phenomenal.

      This is a simple thing to apply when stuck on big projects: break it down, and get a feedback loop going that brings you towards the first milestone.


      Take-away


      Always look with a fresh view to your current way of working. How much better would things get if you got feedback sooner? That's where the opportunities are.




      P.S. when finishing this I realized I could have written the same things about other cost factors. Time is a cost, but so are money, energy and materials. The framework could be the same and very similar observations could be made.

      Comments

      Post a Comment

      Popular posts from this blog

      "Security Is Our Top Priority" is BS

      First published on
      A couple of years ago I was asked to give a conference talk about software security. Well, actually I wasn't really asked, my company bought a sponsorship package with a speaker slot and I replied to the internal email asking for volunteers  🤣  Anyway, while preparing my talk, I realized a couple of important points about security that have not left my mind since: Security is limitless . You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc. The needs of security are opposed to the needs of a convenient user experience . Improving one typically hurts the other. Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that...
      Read more

      AI programming tools should be added to the Joel Test

      First published on
      Here's a wake-up call to all CTOs: AI programming tools are getting freaking amazing and if you don't allow your teams to use them somehow, it will bite you in the ass in a couple of years. You will be slower and you will lose your best people. The infamous Joel Test is a list from the year 2000 of 12 things all great software companies do. Since then most companies have implemented Git and CI/CD, checking of three items, so we have some space left in the 2024 update ;) I believe "Do you allow your developers to use AI assisted development environments?" is a necessary addition. I get that you don't want your source code to end up on some OpenAI / Microsoft / Github server somewhere, sure, but find a way to use your own models or learn to live with it. Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money. So why do I think developers need AI programming tools? I...
      Read more

      The long long tail of AI applications

      First published on
      Image
      Last year multiple companies asked me for advice. " We are evaluating this AI powered product, but do you think it makes sense at all? It seems a bit niche and we think ChatGPT might make this entire thing obsolete soon. " Long tail - credit to Wikipedia My answer so far has always been: "No, bare LLMs are not going to compete with this product." I think that people are failing to understand the distinction between different classes of AI companies. I see it like this: Foundational AI - creating models like GPT-4 (text), Sora (video) etc. Applied AI - using existing models to create smart applications. Note that there are also companies not focusing on AI - they will start lagging behind, let's forget about them ;) There are orders of magnitude  (!)   more companies that will deal with Applied AI than Foundational AI, and they will be very busy for decades to come. Here's why: To get the most out of LLMs, you have to ask the right questions. LLMs have acc...
      Read more