Measuring Security – Ken Connell – Medium
I was recently speaking with a team member of a Cyber Security Operations Center (CSOC) discussing cyber security performance metrics. This individual made an interesting statement saying they were a lot more productive this week than last week because they’d closed more “tickets” from their workflow. My response was, “so if closing tickets equals productivity, how many tickets do you have to close to thwart off a cyber-attack?”
That was obviously a trick question but led to further discussion as to the value of productivity-based metrics and how they should be accurately leveraged to depict an organization’s success rate for achieving their security goals. This starts with an accurate picture from leadership as to what their security goals and priorities are so team members can better report metrics that will accurately predict when teams will reach their finish lines for key initiatives. If an organization is focused on the wrong security metrics, then it can lead to teams performing work that doesn’t contribute to a successful security posture.
Leaders should clearly define expected security metrics to better measure a team’s effectiveness, which will lead to a more accurate picture of performance. The metrics should be defined in a way that every team member can easily interpret the defined conclusion and their course of action to work towards. Security metrics should measure progress towards a shared goal and never just focus on performance alone (which is different than effectiveness).
An example I can relate to is from my time serving in the U.S. Army in Bosnia and later Iraq. Commanders desire metrics to report both the effectiveness and performance of their missions. A common approach I witnessed was counting the number of boxes of food we’d successfully delivered during humanitarian assistance missions. The logic being that the “second and third order effects” would be that the population was less hungry/thirsty, happier, and would become friendlier to our presence (aka: “winning the hearts and minds”). However, tracking the number of boxes delivered provided an inaccurate picture of the population’s sentiment towards us. They were less hungry and thirsty, but this performance metric didn’t correlate with us getting shot at any less (which would’ve provided a more accurate metric of our effectiveness). Successful metrics should combine both measures of performance and effectiveness.
A better approach in that example to identify an accurate metric might have tied together the goal (improved morale and happiness in area). Combining a calculated layered approach like this would’ve provided a better picture of mission success and the knowledge the Commanders really needed to plan future courses of action.
Another key ingredient for effective metrics is to make sure that they can be easily interpreted across multiple teams. Given each team is operating from a different “field of fire,” they each have a different focus and self-derived understanding of success based on their mission sets. To make metrics valuable for both teams and key stakeholders to consume, metrics must clearly illustrate when security operations are succeeding and when they’re failing. Commonly this results in the stop-light color-coding approach using Green, Yellow, Red. However, I’d argue that approach is too simple on its own and requires additional context to show when things are good, bad, and worse. This metric should also be translatable across teams for them to see where they play a part in mission success and should not require having to correlate other metrics and data points for interpretation.
An example can be illustrated by a security operations team seeking to reduce the response time for false positives. By only measuring the number of false positives detected in each period, we’re missing a key measure of effectiveness because it doesn’t provide what number would show that operations are improving. Reporting the number of false positives reported alone does not provide data on the impact to the team’s workload and ability to focus on other areas of security. A better approach might be to measure the time spent by the team on each false positive and then calculate it with a total percentage of working time available for a team. If a resulting calculation indicates that a team is spending most of their time on false positives alone, then we have a clear picture of an ineffective security operation that a leader can better make course correction decisions from.
Once security metrics are interpreted in a way that defines the “So what” instead of a measure that doesn’t tell a key stakeholder the importance of the data, leaders can shift their course of action to increase their security posture. Military commanders seek the same “So what” information to base battlefield decisions off from their staffs. When a new Lieutenant stands in front of a Commanding General for their first “BUB” (Battlefield Update Brief) and simply reports stats calculated from a spreadsheet of battlefield incidents for that day, they will be quickly “corrected” (and possibly dismissed completely) leading to a realization that they’ve overlooked key measures that would help the Commander make informed decisions based on actionable intelligence. Just like military Commanders, organizational leaders expect measures of effectiveness that provide more than just data points. They want to be able to make informed decisions that will make an organization more secure. This example is commonly referred to as the “DIKW pyramid” (Data, Information, Knowledge, Wisdom).
Metrics should effectively convert Data into Information (Facts+ Second and Third Order Effects) that results in Knowledge (Actionable Intelligence) and Wiser leaders. Once effective metrics are clearly defined, then teams can translate steps into a course of action.
The examples I outlined here are geared towards security teams, but the key takeaway is that quality measures of effectiveness can be applied across any situation. These metrics should measure the effectiveness of a team towards achieving an organization’s goals, translated across different groups, resulting in a clear understanding of what steps should be taken to correct a lack of effectiveness. Organizations should consider this when reviewing current security metrics and identify those that are not effectively helping achieve established goals. Once these useless metrics are identified, they can be eliminated, and focus can then be shifted towards more effective measures that will lead towards a more successful security operation.