The Danger of Metrics
I got an email a few weeks ago about my article Operational Metrics: The Next Step in the Evolution of Defensive Cyberspace Operations1 from a researcher studying the same problem. We talked about my article, some of the other writing I have published on this topic, and and then something I briefly touched on in my SOC Metrics series: the dangers of metrics.
In SOC Metrics, Part I: Foundational Metrics, and then again in Individual versus Program SOC Metrics, I explained how poor management could misinterpret metrics to damage rather than improve their organization. This conversation, though, reminded me of Justin Lister’s prescient point, that “Cyber metrics are like Schrödinger’s cat. In that you can’t manage what you can’t measure. But if you measure it then it starts to drive the wrong outcomes.” What gets measured gets managed. Similarly, as Charlie Munger said, “Show me the incentive, and I will show you the outcome.” Numbers might not lie, but the way we contextualize, frame, and present them leaves a great deal of room for biases — inadvertant or intentional — to creep in at each stage. Mark Twain’s famous quote, “There are three kinds of lies: lies, dammed lies, and statistics” sums this up well: we must account not just for lies, straightforward falsehoods, but also “damned lies” or malicious lies, and also the most dangerous category, statistics — those lies that seem most convincing and thus best obscure the truth.
We must measure the right things in the right ways, and we must also manage the right things in the right ways. The danger of metrics is that it opens us up to that most dangerous of scenarios, the one in which we believe we are doing the right things for the right reasons, but in acutality are optimizing for an arbitrary goal disconnected from reality.