Categories
Leaders Letter Newsletter

Leaders Letter 185 – Free Metrics Masterclass From Jeff Bezos

Dear leaders, I have an ongoing loathing of bad metrics. 

Whenever I speak to companies the first thing that is often brought up is performance and then, their associated metric(s). 

My dilemma; is how quickly I tell companies they have bad metrics and often how rotten their OKRs (or equivalent systems) are. 

You will have heard me rant about noise vs signal many times in Leaders Letters. 

Many professionals allow noisy metrics to: 

  • Influence their decision-making process 
  • Confirm their biases 
  • Misunderstand what of the noise is driving significance and which signals are contributing to success or leading to failure. 

Usage Vs Chrun Metrics: In an app company I worked with, the big obsession was usage metrics, there was no metric for successful usage, just if they were a DAU (daily active users), a WAU, or a MAU. 

Churn was almost exclusively ignored. Yes ignored, retention and churn was not something they ever looked at, let alone discussed. 

Imagine if the user opened the app and most days they had a bad experience with the app and ended up using your competitor. 

An example would be Google Maps providing a bad experience and users churned after a couple of uses over to Apple Maps and their daily trips and memories (maps products aren’t just A-to-B products, they help you plan and live important memories) they were arranged and guided by Apple Maps. 

Yes, this does happen and is happening more frequently. 

What I have learnt over the last 22 years is metrics can be good and bad, and the worst metrics are often what senior people obsess over and distress others over. 

Enter Jeff Bezos And His Unknowing 3-Minute Metrics Masterclass On Lex Fridmans Podcast: 

  • Why metrics are mostly good but not great
  • Why metrics can become outdated and mislead companies and their leadership teams (this is common, many ELTs and SLTs never revisit original metrics and understand what I call the metric decay)
  • Why metrics are often just proxies 
  • How the world shifts and metrics do not (and many do not proactively change)
  • Why old metrics are often why day 2 companies struggle  (a day 2 explained by Jeff is “Day 2 is stasis. Followed by irrelevance. Followed by excruciating, painful decline. Followed by death. And that is why it is always Day 1.”)

Watch The Masterclass 

» Scroll down to the bottom of this newsletter for the transcript

This 2-Minute Exchange Is Critical

“The proxy for truth. Let’s say in this case it’s a proxy for customer happiness, but that metric is not actually customer happiness. It’s a proxy for customer happiness. The person who invented the metric understood that connection. Five years later, a kind of inertia can set in and you forget the truth behind why you were watching that metric in the first place. And the world shifts a little and now that proxy isn’t as valuable as it used to be or it’s missing something. And you have to be on alert for that. You have to know, “Okay, I don’t really care about this metric. I care about customer happiness and this metric is worth putting energy into and following and improving and scrutinizing, only in so much as it actually affects customer happiness.”

And so, you’ve got to constantly be on guard and it’s very, very common. This is a nuanced problem. It’s very common, especially in large companies, that they’re managing to metrics that they don’t really understand. They don’t really know why they exist, and the world may have shifted out from under them a little and the metrics are no longer as relevant as they were when somebody 10 years earlier invented the metric.”

The whole podcast is a business masterclass and something you will want to grab your caffeinated drinks of choice and your notebook or notes app and take copious notes to action. 

More, Cheaper, Free & Stagnation

We live in a world of choice, cheaper products, more competition than ever before and in many companies they allow poor leadership and bad metrics to stagnate. 

I have been in a handful of incredible QBRs (quarterly business reviews) and AOPs (annual operating planning), I have also been in some time and energy-draining QBRs and AOPs which many would have paid to leave. 

  • In the best, we discussed the metrics, the actual signals creating real business impact and the actions we were taking forward and the optimisations we would make. 
  • At the worst, we never discussed what the signals were telling us, what the missing links were and how we would truly fix the rot, despite many of us attempting to force the change many executives are happy for BAU and status quo. 

I cannot imagine Jeff Bezos allowing Amazon, Blue Origin or The Washington Post to have bad meetings, QBRs or AOP cycles and not taking competitors seriously and focusing the rest of the exec team around positively obsessing over the customer. 

This week’s focus action is for you to review your metrics, understand which are working to improve your business, which are there just because they always have been there, which are actually limiting your company and the metrics that are negatively impacting you and your business. 

Have a great last week and get in touch if you would like to

Thanks,

Danny Denhard

.

The Transcript (source)

Jeff Bezos

(01:24:36) Well, I’ll talk about… Because I think it’s the one that is maybe in some ways the hardest to understand, is the skeptical view of proxies. One of the things that happens in business, probably anything where you have an ongoing program and something is underway for a number of years, is you develop certain things that you’re managing to. The typical case would be a metric, and that metric isn’t the real underlying thing. And so maybe the metric is efficiency metric around customer contacts per unit sold or something like. If you sell a million units, how many customer contacts do you get or how many returns do you get? And so on and so on.

(01:25:30) And so what happens is a little bit of a kind of inertia sets in where somebody a long time ago invented that metric and they invented that metric, they decided, “We need to watch for customer returns per unit sold as an important metric.” But they had a reason why they chose that metric, the person who invented that metric and decided it was worth watching. And then fast-forward five years, that metric is the proxy.

Lex Fridman

(01:26:02) The proxy for truth, I guess.

Jeff Bezos

(01:26:04) The proxy for truth. Let’s say in this case it’s a proxy for customer happiness, but that metric is not actually customer happiness. It’s a proxy for customer happiness. The person who invented the metric understood that connection. Five years later, a kind of inertia can set in and you forget the truth behind why you were watching that metric in the first place. And the world shifts a little and now that proxy isn’t as valuable as it used to be or it’s missing something. And you have to be on alert for that. You have to know, “Okay, I don’t really care about this metric. I care about customer happiness and this metric is worth putting energy into and following and improving and scrutinizing, only in so much as it actually affects customer happiness.”

(01:27:03) And so you’ve got to constantly be on guard and it’s very, very common. This is a nuanced problem. It’s very common, especially in large companies, that they’re managing to metrics that they don’t really understand. They don’t really know why they exist, and the world may have shifted out from under them a little and the metrics are no longer as relevant as they were when somebody 10 years earlier invented the metric.

Lex Fridman

(01:27:29) That is a nuance, but that’s a big problem. Right?

Jeff Bezos

(01:27:33) It’s a huge problem.

Lex Fridman

(01:27:34) There’s something so compelling to have a nice metric to try to optimize.

Jeff Bezos

(01:27:38) Yes. And by the way, you do need metrics.

Lex Fridman

(01:27:41) Yes, you do.

Jeff Bezos

(01:27:41) You can’t ignore them. Want them, but you just have to be constantly on guard. This is a way to slip into day two thinking would be to manage your business to metrics that you don’t really understand and you’re not really sure why they were invented in the first place, and you’re not sure they’re still as relevant as they used to be.