Mike Swanson explains the history of software tracking and how things became as they are today. Here’s Mike explaning a well known warning—when a measure becomes a target, it stops being a good measure.
One of the most dangerous things about analytics is that they feel objective. A chart is a chart. A number is a number. They have the aesthetic of truth.
I’ve always liked this quote by William Bruce Cameron (often misattributed to Albert Einstein):
“Not everything that can be counted counts, and not everything that counts can be counted.”
Metrics don’t measure reality. They measure what your product currently makes easy.
There’s a well-known warning about this, often summarized as: when a measure becomes a target, it stops being a good measure. It’s commonly referred to as Goodhart’s Law, and the broader point shows up in multiple fields, because it keeps happening to humans in systems with incentives.
When I was at Microsoft, a team wanted to remove a feature because “the analytics show that nobody uses it.” If you looked at the UI, though, that feature had been moved deeper and deeper over time:
- it used to be easy to find
- then it moved into a menu
- then into a submenu
- then into a settings panel
- then behind an “advanced” section
- then it was basically invisible
Of course nobody used it!
The analytics didn’t prove the feature was unwanted. The analytics proved that we buried it.
Even worse, once a metric becomes a target, people get promoted for moving it. That doesn’t require anyone to be malicious. It just requires incentives and a dashboard.
naveegator.in 