Checklists -The Preserve of the Intelligent
As the New Yorker says “If something so simple can transform intensive care, what else can it do?”. Dennis Groves sent me this article a week ago and I read it twice. Each time I couldn’t stop myself thinking about how many people in the information security industry shun checklists and considering why this is. If fighter pilots, A&E surgeons and astronauts use them why don’t we see the value?
Mushroom Syndrome - In the land of the blind, the one eyed man is king and for a long time some people have claimed security was a black art. Mushrooms are kept in the dark and fed cow secretion.
Misconception - Checklists don’t fly fighter jets, skilled pilots do. Many security analysts argue that you can’t create a checklist for code review or penetration testing. This is a moot argument. Of course you can’t but that’s not what a checklist is. It’s a tool to help you make sure you cover the critical issues when under duress. How many of the 93.75% of undiscovered bugs may have been found if excited researchers had been prompted to checklist to cross-check a new finding against similar easy conditions?
Quality - I doubt the quality of the checklists used in the industry today share much resemblance to the quality of checklists used by an airline pilot. I have seen some shockers.
It doesn’t need to be this way. I am going to share some InfoPath forms I have been using in the coming months.
Yesterday I was reminded just how powerful clear and concise instructions can be when I built Alpha Rex with my 7 year old. Yes my 7 year old (who is a genius but then again I am biased) built a real robot by following this visual checklist.
I suggest that checklists are the preserve of the intelligent (without egos).
February 17, 2008 at 1:22 pm
That robot checklist is awesome, but I rather than making this an argument for good checklists, I would say that this is an excellent argument for the power of well-done technical documentation. Documation is something that most people in our industry abhor, because that is not what they went to school for. Programmers need to program. security peeps need to secure, documentalists need to document and librarians need to librarify(?). The latter two are common position in “old-school technical” professions, such as aviation, engineering, etc. I still need to see many of these job titles in Information Security.
February 17, 2008 at 1:23 pm
(apologies for the typos in the above comment; my toddler was “helping”)
February 17, 2008 at 5:59 pm
(specifically talking about testing here - as that’s my field, but appropriate to other areas as well)
Um, if a methodology is’t a (form of) checklist, then what is it?
You need a checklist/methodology, or you just end up doing ad-hoc stuff. There’s nothing “wrong” with an ad-hoc approach, as it can (and does) find thing that you wouldn’t usually, but it has to be timeboxed.
I would be very surprised to know that people doing testing, or configuring machines, etc, are not using some checklist or methodology. When it comes to system design/programming on the other hand, I’ve too often seen these things left until the last moment.
February 20, 2008 at 12:51 am
@ MikeA:
Excellent point. Exploratory testing assumes a lack of a checklist (i.e. test cases) and allows the testers to work by building a test charter while learning the application. It doesn’t mean that other testing methods can’t be used. Exploratory testing also sets itself apart from ad-hoc testing, but maybe that is a separate argument. I also don’t feel that it has to be “timeboxed” unless of course you are referring to the idiom that humans repeat themselves every half-hour. In this situation, it’s best to set aside time buckets for Exploratory testing, with test case improvements in between that time.
I’m also a fan of the Think Aloud Protocol and pair-testing. The testing reputation system put forward by Ramya Venkataramu and Ryan Gerard while at Symantec is one of my favorite concepts for this type of testing.