Step Eight – How Do We Know When It’s Working? (Defining Metrics)

This blog walks cybersecurity professionals through the logical steps to create a practical and actionable cybersecurity strategy.

You must be able to measure the successful realization of your strategy as it is operationalized.

You will recall I said earlier that the cybersecurity strategy should be considered a living thing that should be reviewed and revisited on a continual basis to ensure it is 1) still relevant and 2) doing what you want it to. To help accomplish #2, you need to establish realistic and tangible metrics that can tell you how well your strategy is being executed – your vision realized.

But what kind of metrics would be appropriate for measuring the successful operationalization of a cybersecurity strategy? Let’s think about that. When working to define your metrics, here are a few things to consider.

Don’t Confuse the Strategy’s Metrics with Your OKR Metrics

There is a fundamental difference when you are considering what metrics will allow you to understand how well your strategy is being realized as opposed to the initiatives and projects being executed to accomplish the strategy. It is this:

Cybersecurity Strategy Metrics allow you to determine how well and/or effectively the vision and objectives of the strategy are being realized over the lifetime of the strategy.

Objective and Key Results (OKR) Metrics (measurements) will allow you to determine how well or effectively your individual projects, initiatives and programs are being delivered and whether they are achieving the stated requirements. This is oriented on a project-by-project basis and is basically tactical.

Focus Metrics on Measuring the Successful Realization of the Vision and Goals of the Strategy

When determining your metrics, be sure to focus on the long-term objectives of the strategy and what it is trying to achieve for the organization. They have to address the high-level requirements the organization needs cybersecurity to accomplish and do so in a manner that is easy to understand by non-technical audiences.

Here are a few examples.

• If the organization has, as one of its strategic objectives, the need to improve its ability to maintain their operational capabilities, it can develop the following metric:

Enterprise Resilience – Enhance business continuity and disaster recovery preparedness capabilities. Measure the effectiveness of these programs with respect to their ability to allow operations to continue and recover in the face of incidents and/or disasters.

• If the organization has, as one of its strategic objectives, the need to improve compliance with policies and greater consistency in operations, it can develop the following metric:

Policy and Standard Exceptions – The number of exceptions and exception extensions against current policies and standards will be measured to determine formal compliance with published organizational requirements.  The expectation is that, as the organization moves to align its administrative and operational activities with established policies and standards, the number of exceptions and extensions should lower on a continual basis.

• If the organization has, as one of its strategic objectives, the need to improve user involvement in securing the organization’s information ecosystem, it can develop the following metric:

User Engagement – The organization will measure the effectiveness of efforts to raise user awareness of cybersecurity best practices and their collaboration and partnership in both securing the organization’s assets and safeguarding customers’ privacy. This effort will also work to establish their general level of engagement with information security through such metrics as, self-reporting of issues, surveys on security events and phishing response rates.

Other metrics might include, depending on the organization’s priorities:

• Reduction of Risk
• Improved Threat Telemetry Gathering and Analysis
• Service Coverage
• Vulnerability Management Improvement
• Application Security Metrics
• Network Defence Metrics
• Delivery Against the Strategic Plan Roadmap

You will notice that the above examples do not outline specific measurements; only that they are to be determined. This is intentional as the strategy is pointing out what must be measured and why. How it is to be measured may be a considerable effort and would likely require the collaborative input of a number of resources. There may be technical and operational requirements in monitoring and scaling the metrics. Therefore, metrics are to be determined during the execution of the strategy and the how does not necessarily belong in the plan document.

The main message here is to focus on those metrics that clearly communicate whether the cybersecurity strategy is succeeding in accomplishing vision; the big picture. Ultimately, make sure they are clearly defined, can be tangibly monitored and empirically measured.  

Finally, as with many parts of a cybersecurity strategy, stand ready to adjust the metrics as the need arises. The measurements may need to change as the strategy evolves or changes and should not necessarily be set in stone. As the strategy moves forward, make sure the metrics you have put in place are providing useful intel or whether they need to be:

• Augmented;
• Adjusted; or
• Retired.

Thank you for your time. Be sure to stay tuned for the next posting in this series: Cybersecurity Strategy Step Nine – How Do We Get There? (Defining the Operational Roadmap for the Strategy).