Always Optimize the Feedback Loop

Speed, I am Speed. -- Lightning McQueen

Some thoughts about the impact of feedback loops in startups and software.

Parts:

• Intro
• Framework
• Observations
• Take-away

Intro

Feedback delay is the time between doing something and seeing whether it was successful.

If the feedback is slow, we have a "lagging indicator". It becomes hard to link causes and effects, and in consequence it becomes hard to optimize the performance. Fast feedback makes things easy.

How quickly the feedback arrives is critical. If it takes milliseconds to seconds, it works at the speed of the brain and enables a state of flow. If it takes decades to centuries, it needs a multi-generational organization like a government or organized religion to run it. Somewhere in the middle is the realm of persons, teams and companies.

A framework of feedback time

Here's a quick table of feedback times, in three groups, with each roughly an order of magnitude difference.

C is the level where (large) organizations can run experiments without seeing the results for a long time

• C4 centuries
• C3 decades
• C2 years
• C1 months

At B a person or team can have the patience to execute this

• B4 weeks
• B3 days
• B2 hours
• B1 minutes

A is the speed of the brain, the feedback arrives so fast that it feels effortless.

• A4 seconds
• A3 100+ milliseconds
• A2 10+ milliseconds
• A1 1+ milliseconds
  

Observations

With this mini framework in mind, here are a couple of observations.

1 - Startups

Startups are viable if they do something 10x better. Each of the steps is a 10x improvement, so if you manage to create a solution that brings feedback time down one step on the ladder, you have a viable startup. At least according to Peter Thiel.

2 - Faster is always better

Faster feedback is always better and never worse, but there are diminishing returns. E.g. when I'm typing and it takes 4 milliseconds to render to the screen, it's already imperceptible, and improving it more offers no benefit.

When looking to improve something, check what level it is at right now and what would count as imperceptible. The difference is the space that a new product or service could enter.

For example, in a regulated industry I will get feedback from an auditor at C1 (a couple of months) but my life would be so much better if they replied at A4 (seconds) and I can have a dialogue. So: there's an enormous opportunity here for a regulatory body that promises responses in B1/B2 (see e.g. Scarlet) or for a group of experts you can chat to instantly (OpenRegulatory Slack).

3 - Multiple orders-of-magnitude improvements shake the world

Improving by 1 step is great, but something weird happens when we take 2 steps down the ladder or more. Things that were previously impracticable or downright impossible, now become in reach.

When jetliners were invented, we could travel around the world in B3 (days) where in 1930 it still took my grandfather C1 (months) to go from Indonesia to The Netherlands. Jet travel has certainly changed international business, politics and tourism.

A recent example are LLMs. Getting a book summarized would take a human B2 (hours), and ChatGPT does it in A4 (seconds). By making it two orders of magnitude faster, we just went into flow-state territory!

When a new technology makes multiple orders of magnitude improvements, the world needs a couple of decades to adjust to this new reality.

4 - Software vs Hardware

Most software development features are in B, while hardware is in the lower Cs. They are wildly different industries because of this, with different approaches to QA.

Software developers that go into regulated industries see their feedback time going from B1 (when automated tests tell me there is a problem) and B3 (when customers tell me if they like something) to C2 (the regulators will review your documentation and give you feedback in about a year). Talk about culture shock!

5 - Hunt for the feedback loop

Whenever I am stuck on a problem, I have bad tendency to get blocked and not knowing where to start. When I notice this happening, I've learned to try to find something to do that works towards the goal and that gives me feedback the fastest.

In programming there are feedback loops that will always get me into a hyper productive state:

• Getting compiler / linter errors in my IDE. When I've refactored code or when I introduce a linter to help clean things up, I get a bunch of new errors that I just have to fix one by one.
  
  
• I execute the program and get an error, then I fix it and go to the next error.
  
  
• Getting CI test results (e.g. speed of the program, memory usage, code coverage). Changing something and seeing the result of my change gets me feedback pretty quickly, but CI is often in B1/B2 (minutes/hours) and unfortunately frequently gets me out of flow state.
  
  
• Create a to-do list of things to finish, and try to get it to 0 as quickly as possible.

SpaceX famously achieves rapid progress through rapid feedback cycles compared to the upfront design work at e.g. NASA.

Their long-term goal is to colonize Mars, which is clearly in C4 / C3 (centuries / decades). They won't know if this will work until it's done. Those kind of grandiose projects have a bad habit of not getting started. SpaceX brilliantly broke the problem up into multiple milestones, and got a feedback loop going towards milestones 1 - launching Falcon 9, 2 - making the boosters re-usable, 3 - building a bigger rocket which is entirely re-usable.

Their results are phenomenal.

This is a simple thing to apply when stuck on big projects: break it down, and get a feedback loop going that brings you towards the first milestone.

Take-away

Always look with a fresh view to your current way of working. How much better would things get if you got feedback sooner? That's where the opportunities are.

P.S. when finishing this I realized I could have written the same things about other cost factors. Time is a cost, but so are money, energy and materials. The framework could be the same and very similar observations could be made.