{"id":6907,"date":"2015-01-07T17:45:00","date_gmt":"2015-01-07T17:45:00","guid":{"rendered":"http:\/\/intercom.com\/blog\/?p=6907"},"modified":"2020-07-30T13:02:25","modified_gmt":"2020-07-30T12:02:25","slug":"meaningful-growth-vs-metric-manipulation","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/meaningful-growth-vs-metric-manipulation\/","title":{"rendered":"Meaningful growth vs metric manipulation"},"content":{"rendered":"

What gets measured gets done, no matter the cost. But there is a genuine danger in reducing product onboarding and design to an exercise in metric manipulation.<\/p>\n

The last decade has brought us hundreds of analytics products. They all gather easy-to-collect data, and then render it in \u201cbeautiful to look at\u201d visuals. Of course, no one checks if this \u201ceasy to collect\u201d data is actually informative. Or if the \u201cbeautiful to look at\u201d visuals are actually useful. But even when they are, and you have an accurate well defined funnel, playing whack-a-metric will inevitably lead you astray, blinded by funnel-vision.<\/p>\n

Because what gets measured gets done. That phrase is often used to imply something like \u201cNothing will happen here until we have a clear metric<\/i>\u201d which is often true. Ambiguous phrases like \u201cwe\u2019ll take a look at it<\/i>\u201d, \u201cwe\u2019ll see if we can improve<\/i>\u201d, or \u201cwe\u2019ll tackle the low hanging fruit<\/i>\u201d are the bane of any product team. It\u2019s impossible to tell if anything actually happened as a result, whereas numbers don\u2019t lie.<\/p>\n

But numbers are selective with the the truth. When you instrument any flow, feature, or ratio, there will be a way to game it.<\/p>\n

The Rat Effect<\/h2>\n

When Hanoi was under French colonial rule, they had a big rat problem. They tried to solve it by rewarding people for killing rats. To prove you had killed a rat, you presented it\u2019s tail. What happened next is a good example of bad proxies. Citizens began to breed rats, cut their tails off, claimed the prize, and used the tail-less rat to breed more rats. The solution was backfiring. Yet at the same time the government was reporting \u201crecord progress\u201d in rat extermination, as their metrics were through the roof. This became known as the Rat Effect, also known as the Cobra effect<\/a> for a similar incident in India.<\/p>\n

In both cases what got measured got done. And what got rewarded, got repeated.<\/p>\n

All metrics, even if you hand pick them, are a proxy of user behaviour. Your project management tool might use \u201cprojects opened\u201d to measure if users are getting value from the product, and that could well be a good proxy, but all proxies are leaky, and susceptible to gaming.<\/p>\n

When we say \u201cgaming\u201d here, we mean influencing the metric in a way that doesn\u2019t similarly influence what it\u2019s a proxy for. For example, you could make \u201ccreate a project\u201d a mandatory step for every single user post-sign-up, or you could even do it automatically for them. The metrics will go up, but it\u2019s not clear users are getting more value. You haven\u2019t done anything substantial. It reminds me of the British sitcom where police officers boast they\u2019ve reduced crime simply by legalising burglary.<\/p>\n

Ev Williams uses \u201cTotal Time Reading\u201d as a metric for Medium<\/a>, but points out that there\u2019s lots of ways to influence it that don\u2019t help the cause, for example worse typography would easily be an \u201cimprovement\u201d by that metric. Similarly I often joke that the editor Vim has the best growth hack of all-time, they boost daily active users by making quitting the editor ludicrously complex<\/a>.<\/p>\n

If you want to create meaningful growth your efforts are best spent in one of three ways, removing friction,\u00a0relocating friction, or pointing users in the right direction. We\u2019ve talked a lot about how pro-actively helping users \u00a0increases conversion<\/a>, decreases churn<\/a>, and increases customer satisfaction<\/a>, so let\u2019s talk about friction. Specifically removing & relocating it.<\/p>\n

Removing friction<\/h2>\n

The best metrics-driven<\/a> meaningful growth will always come from identifying, removing, and relocating friction. If someone is at step 3 and fails to get to step 4, friction is usually what stops them.<\/p>\n

Why did the user not get to step 4?<\/p>\n