In response to yesterday’s “Sad State Of Metrics” Rybolov writes:
Technical things are easy and cheap to measure, usually the tool gives you all sorts of statistics. However, we usually have those problems figured out already–we need metrics for the stuff we don’t know about yet, and that’s where the concept of metrics falls apart.
And then points us to his excellent post on Puzzles & Mysteries. In it, the “that” which “we don’t know about yet” he describes:
…we have the mysteries: 0-day attacks, covert channels, and the ever-popular insider threat. Just like a well-established military has problems understanding the mystery that is movement to contact, information security practitioners have problems responding to threats that have not been well-defined.
Clearly he’s correct. In Rumsfeldian - these are the known-unknowns (sorry). We know that at any given point in our controls framework - the distributions may shift and we may end up on the wrong side of the Threat Capability vs. Control Strength battle. The good news is that we do know the “points of attack” or attack vectors that they may use to get to us.
ENTER RISK ANALYSIS
One of the cool things about FAIR is that as a framework for stochastic analysis, it works well at various “levels of abstraction”. Jack has a saying that I’m fond of - analyzing risk at times is like trying to determine the surface area of Long Island. If you go to Google Maps, and enter Long Island, you can use the little “zoom” thing to move in and out to get different “levels” of Long Island. We can even get street views. We can look at risk in a similar manner (well, maybe not the street views thing). The trick is determining which “zoom level” is appropriate to obtain useful analysis.
Now I can’t reveal your known-unknowns to you. But what I can do is encourage you to perform FAIR analysis at a fairly high level of abstraction (”zoom out”) in order to think hard about frequency and impact of known-unknowns at various points in your “surface area”. Because we usually know those key points of attack - we can aggregate asset variables into an overall “business process” estimation for TEF, Vulnerability, and Probable Loss Magnitude. This high-level view will allow you to begin to identify critical areas to focus on. Now further analysis may be needed as the mysteries you have turn into puzzles - as analysis helps you move towards a state of knowledge - allowing you to then “zoom in” one more level and find your new known-unknowns.
ENTER CONTROLS ANALYSIS
Now you’re not going to be able to prevent a zero day, and at some level you’re going to run up against unknowns that prevent you from moving to the next level of abstraction in your analysis. But what you may find is that you can reduce risk or loss by increasing your ability to detect and respond to those types of threat actions at those areas you want to focus on. Recall:
- If we have 100% effectiveness at prevention, we don’t really care about our ability to detect and respond, do we?
Similarly,
- If we have 100% effectiveness at detection and response, we don’t really have to worry about prevention!
In the real world, we’re not 100% efficient, but by identifying which of the 3 (prevent, detect, respond) we’re good at, we can focus resources on the others in order to; a). have more control - reducing risk, or b). reducing the impact of an event - reducing loss. Recall our opportunities to create value statements from yesterday.
ENTER RISK MANAGEMENT
What you’ve just done is taken a look at your ability to make good risk decisions, (state of nature: risk decisions) and identified where you’re lacking information to make good risk decisions (state of nature: risk posture). We’ve then used risk analysis to find where you need to move from state of nature to a state of knowledge, and implemented controls to help develop that state of knowledge (from state of nature: risk posture -> state of knowledge: risk posture). That’s almost a discreet risk management task! The final ingredient is to develop assurance controls to make sure that you’ve correctly identified your capability to manage risk and at least keep that capability state current (moving us from state of nature: risk decisions -> state of knowledge: risk decisions).
ENTER METRICS
Another of Jack’s sayings is that this whole journey we’re on concerning risk and risk management is like untangling a giant ball of string. Working on various knots and untying them in order to develop frameworks in which we can operate - identifying the factors that create the mystery and then moving that mystery into a state where it simply becomes a puzzle. If you’ll allow me the analogy - we’ve got the picture on the box. Finding the factors is like identifying the border pieces and creating the frame. Once we’ve identified the factors, what’s left is determining the mathematical relationships that best mirror reality (scientific method), and the metrics we need to make those equations work. That’s the more difficult act of filling in the puzzle. It’s not easy, but it’s a heck of a lot of fun.
If you like puzzles, that is.
Tags: metrics, compliance, information risk, information, risk, risk management, risk_management, information, security, information security, information_security, governance, compliance, enterprise_architecture
(via RiskAnalys.is)
June 20, 2007, 9:11am Comments
