Measuring Cyber Security Effectiveness

Measuring and understanding cyber security effectiveness – where do you start?

“You can’t manage what you can’t measure.”

Peter Drucker

This classic quote from management guru Peter Drucker is equally applicable to the measurement challenge we face in information security.

If we can’t (or don’t) measure, how can we identify if we’ve been successful with our security initiatives? For example, is a lack of securityincidentsan indication ofsuccess? If so, how can we demonstrate it? Likewise, without evidenceof what isworking (and where) – how can we justifyoften significant security expenditure- or indeed,make effective security decisions?

In this article we’ll take an introductory look at the use ofmetricsto measurecyber security effectiveness(sometimes also referred to as ‘security maturity’)withdashboards and benchmarking. We willalsoreferenceexisting frameworks and modelsand providelinks for you to explore in more depth.

But first, let’s start with a definition that some might find helpful…

Measurement vs Metrics – what’s the difference?

The US National Institute for Standards and Technology (NIST) defines a measurement as something that is quantifiable and observable. Whereas a metric is defined as something supported by measurements that intends to facilitate decision making, improve performance and accountability.

Many experts agree that metrics are probably the most usefuldata pointsat ourdisposal.Thosereaders with experience in cyber securitymay like to dive deeperinto the NIST Framework (2017) (although still in its draft form), whichexpands on the topic by identifying metrics (in section 4)to enablethe measurement of cyber securityeffectiveness.

Similarly,theCenter for Internet Security(CIS)Top 20Critical Security Controls (CSC) libraryincludes a measurement download that enables users to identify key information to help them track progress.

Here atAscentor, we are fans of the NIST Framework andCISTop 20CSC – we have had excellent results working with them and wouldtypicallyconsider them fornew projectswhere appropriate. Forsome of you, a lighter touch may be helpful, so we have also included a selection of alternative options below.

Report what you should, not what you can

In a similar vein to Drucker’s thoughts on the need for measurement,PhilCracknell(CISOofHomeserve), put the case for metrics while speaking at the 2017 Cyber Security Summit and Expo. He said“businesses are waking up to the fact that they need metrics and risk indicators that our board, audit committees and non-executive directors are able to understand”.

He suggested that businesses should adopt a “report what you should, not what you can” approach and that metrics “can demonstrate effectiveness, measure exposure and agility, testorganisationculture, pinpoint responsibilities and highlight levels of investment”.

Measurement: Two approaches and what to include

There are two common approaches to measurement – a dashboard of metrics, and benchmarking (whereorganisationscompare their relative performance to others).


Manyorganisationshave adopted a ‘dashboardapproach as a means of communicating with senior management. Meaningful metrics need to be quantified in terms of time, moneyand/or risk level, so suggestions for what a cyber security dashboard might include are:

  • Financial losses due to security breaches
  • Damage to reputation and trust
  • The estimated cost of lost customers (see our article on the cost of the TalkTalk hack )
  • Down-time due to business disruption
  • Time taken to deactivate former employees’ access
  • Frequency of attempts to access servers or applications byunauthorisedusers
  • Types of vulnerabilities discovered
  • Time taken to mitigate vulnerabilities
  • Patch management – the regularity of patches and system updates and any errors detected (see our article on when patching goes wrong )
  • Pass/fail results for employee information security training initiatives

For those following the NIST approach, the 2014 NIST Framework suggests that a dashboard could be aligned to their NIST Framework Core Functions .These consist of five concurrent and continuous functions: Identify, Protect, Detect, Respond and Recover.

Once your data has been gathered, what is it telling you? Look for data that might not be what you were expecting or if there are any inconsistencies. How does it compare to earlier dashboard measurements of the same data? And, what action will you take? Information informs action – or at least it should.

A2015 Tripwire magazinearticleasked a number of security industry experts for their tips on what they recommend a powerful dashboard must have . It is a worthwhile read and goeswellbeyond what can be covered inthisblog article.

2) Benchmarking

Benchmarksare great forhelpingorganisationscompare their relative performance.The problem withusingbenchmarkingto measure security effectivenessis the difficulty of identifying suitable metrics from comparableorganisations, and even thenthey might not have theirs right.Consistencyof benchmarking metricsisa real challengeforinformation security.

Therefore, for information security, benchmarking by direct comparison with otherorganisationsis probablyfutilein most cases. A viable alternative, which we support, is to use a form of maturity model as a benchmark.

For cyber security, we like the Cybersecurity Capability Maturity Model (C2M2) .C2M2 was developed by the US Department of Energy as a mechanism to improve maturity of cyber security for the US energy infrastructure – but it may be used by anyorganisation. It defines ten domains for analysis and four maturity indicator levels (MIL 0, 1, 2, 3). Self-assessment against the model is supported by an accompanying toolkit, which also generates a useful report that includes graphical results and a gap analysis.


With the plethora of frameworks, methodologies and guidelines, it is hard to know where to turn when it comes tometrics andmeasurement.While each approachcan becomprehensive, no single one gives you everything you need. This bringson theadded headache ofa jigsaw approach where you may have tochoosemore than oneapproachandthenworkout how to fitthingstogether.

For smaller businesses, a pragmatic set of cyber security measurementsthat dovetail with your corporate dashboard is a good place to start. You can get more sophisticated as your business grows and your cyber capability and knowledge mature.

Our workwith major corporations and public organisations (where a more robust approach is needed),has led us to review and assess the best-known approaches and tools as well as some of the more obscure ones.From that experience,wefavourthe complementary NIST Framework, CIS Top 20 CSC and C2M2 trio.

For further information

If youd like to discuss how our consultants can advise on any aspect of Information Assurance and cyber security, please contact Dave James, MD atAscentor.

If youd like to make sure you get all our top content, news and views this year, why not sign-up to our newsletter or follow us on Twitter or LinkedIn .

Email: [email protected]

Office: 01452 881712


Other posts you might like

Seven steps to designing a resilient Cyber Security Programme

Convince your board – cyber attack prevention is better than cure

Ransomware attacks – don’t let them happen to you