More On Checklists
Alex Hutton posted this follow up on my first post about checklists. He is of course spot on. Checklists in my humble opinion can provide a State of Nature, but can’t provide a State of Knowledge or a State of Wisdom (nice phrases). They certainly don’t do computation or analysis but what they do is frame a set of activities you might do in order to consistently obtain the State of Nature; from which you can determine a State of Knowledge and Wisdom. That’s a very important function and in my humble opinion the importance of it is being over looked by the general perception that checklists try to do the analysis work (so the expectation of what a checklist is doing fails) and quite frankly by the crappy checklists most of us have to deal with today. PCI (yeah I know I promised I wouldn’t mention the three letter acronym in public again and let it pass but I just can’t help myself) is a classic example of this. Many of the numbered items are so ambiguous that you can easily pass or fail depending on your intent. I am sure everyone has seen the approval letters floating around on distribution lists signing off instantiations of Foundstones Hacme Apps and OWASP’s Web Goats as secure and PCI compliant for instance.
Lets be clear here. “Passing a checklist doesn’t mean that you don’t have a problem, while failing one is a strong indicator that you do”.
Checklists are about human patterns. When people glance in the rearview mirror when driving a car every few seconds they are more aware of their surroundings and so less likely to have an accident. That’s a fact, period. This is why driving examiners (in the UK) have a checklist to validate that the driver is exhibiting this behaviour. Would you prefer he got back to base and then tried to remember all of the characteristics that he personally thinks exist in a good driver (and guess the ones he forgot to record). Of course not: I hope not any ways! A checklist can capture a set of patterns that are agreed up by a group and allow teams to check against them. They don’t have to be paper or physical but usually are until memorized. Even then if you read enough psychology books you’ll appreciate that people change things in their sub-conscious all the time without letting their conscious memory know, meaning the pattern gets out of sync unless it’s stored outside of the brain and referred to or updated. “Of course I was going to do the lawn today dear, I just ……..”. Neuro Linguistic Programming or NLP is fascinating if you haven’t explored it.
In most cases checklists should be written to allow people to easily make clear and simple observations or follow a specific path with specific outcomes. Checklists don’t have to be traditional word lists. Visual aids are superb. Here is a genuine one from a flight checklist to illustrate that point.
“Before raising the fuselage, check the hydraulic pressure gauge shows green”.
Many pilots have been killed by raising the wheels on their planes as they take off, never to be able to lower them and therefore safely land again.
All I know is this. I have seen thousands of technical security assessment reports from hundreds of security consulting companies. The deviation in what they did and didn’t look for is simply incredible and there is significant room for improvement. We will never codify a set of complete activities into a checklist but I do believe that well written, well designed checklists can play a significant supporting role in improving the base consistency and quality of many security processes in the future.
