ISM Community Top Ten – Manage with Facts and Numbers
Another guest article from Tim Smith (video here) to support the ISM Community Top Ten. Leave comments for Tim in the comments below!
Information Security is quite unique in its avoidance of quantifiable metrics to justify its position. Although this is changing, it’s still an oddity almost solely reserved for Information Security. Sometimes it’s perhaps looked on as being too hard. If you compare it to building software then maybe it is trickier – metrics in software coding could be defects per thousand lines of code etc – not much software is built without those metrics and they are used mainly to track the quality of the product.
There is now, however a very real requirement to measure security metrics. Some of these are driven by increasing regulatory controls, financial requirements and also general organisational reasons. In the US, there are Laws that actually cite the requirement for security performance measurement such as the Clinger-Cohen Act and the Federal Information Security Management Act (FISMA). In general terms, there is a move at both the Government and Corporate level to provide metric data on the effectiveness of an organizations security controls in general and in particular where an Information Security
Management System (ISMS) has been adopted. This will only become more topical with the uptake of Sarbanes-Oxley and other applicable Governance regulations.
This has received further emphasis with the imminent (ish) release of ISO 27004 – Security techniques — Information security management measurements, a standard that will be focused on defining metrics for information security management.
Additionally, in July 2003, the National Institute of Standards and Technologies (NIST)) published the special publication 800-55 (sp800-55), Security Metrics Guide for Information Technology Systems.
NIST offers a discussion on creating a metrics strategy, development and approach, implementation guideline is referenced substantially in this dialogue.
The recent frenetic uptake in IT Infrastructure Library (ITIL) and IT Service Management (ITSM), with roots in Capability Maturity Model (CMM), have also laid the foundation and ensured a receptive corporate culture in the light of metrics. The rapid adoption of ITIL has therefore driven the services centric and metric based approach into security as well.
Regardless of legal compliance, it is a pretty good idea to be able to measure the adequacy of in-place security controls, policies and procedures anyway. How else do we know if our existing controls are giving us any benefit or if we have shortfalls?
Quantifiable performance metrics and metrics analysis help to ensure the on-going relevance of an organizations security controls.
This has the benefit of:
· Improved accountability
· Pinpoint specific operational, technical or management controls that are missing or inefficient
· Isolate problems
· Use it to justify expenditure
· Ensure expenditure is targeted where it is required
· Get best value from operational and technical resources
So what are metrics and why do we need them?
Metrics are tools designed to facilitate decision making and improve performance and accountability through collection, analysis, reporting and monitoring the status of measured activities. Metrics should be based on things that can be gauged over time (and compared against a baseline), rather than a one off ‘measurement’.
Metric Types
Metrics are directly proportional to the maturity level of the organization – only metrics that are repeatable and consistent should be used. This, as ever means that the metrics we collate are going to be dependent on your organization so avoid generic metrics or at least modify for your organization.
Maturity Levels
Typically, the maturity of a security program can be determined by the existence and adoption of processes and procedures. The levels outlined below based on the NIST standard demonstrate the analytical process of establishing the organizations level of maturity:
Level 1 – having policies
Level 2 – having procedures
Level 3 – implementing procedures
Level 4 – testing compliance with and effectiveness of procedures
Level 5 – fully integrating policies and procedures into daily operations
As an example, at Level 4 and 5, the metrics concentrate on measuring effectiveness and efficiency. Rather than concentrating on whether procedures/policies exist, they concentrate on how effective the controls are. For example, running a password cracker over passwords that comply with policy giving us the length of time required to crack a compliant password would be useful metric here to determine the suitability of the password credentials.
Additionally, if we can measure the amount of incidents we have had, e.g. root compromises, DoS etc and measure that data against how many security staff we have and their level of training we can gauge the effectiveness of the training and our people (bit tenuous but you get the idea).
We can also then gauge whether performing Risk Assessments and Vulnerability assessments has a positive effect on our security posture over time.
Overall to implement a metrics program we need to;
· Determine what we actually want to achieve from the metrics program.
· Define our maturity level and hence the metrics we are able to accurately gauge
· Generate or find the metric data. This could be found in firewall logs, IDS logs, AV logs or be non-technical using the example of training received by the employees and looking for a quantifiable result. You’ll need to establish a baseline for each metric as it’s the trend over time we’re interested in.
· Make the data meaningful for the particular audience. You’re going to end up with potentially a very technical report – massage the data to suit.
· Continue the metrics program. Start again – the painting the Sydney Harbour Bridge motif again.
Have a look at;
· NIST SP 800-80 – Guide for Developing Performance Metrics for Information Security and its companion standard, NIST SP 800-55 – Security Metrics Guide for Information Technology Systems
· SSE-CMM – Systems Security Engineering Capability Maturity Model
