Mark Curphey - SecurityBuddha.com

A development manager friend in Europe sent me the following email:

…I know you aren’t in the business anymore (?) but how the f&^% do you scope a web application security review? I have asked six firms to provide proposals and after countless wasted hours repeating the same things they are literally 1000’s of percentage points apart in pricing and what they are telling me I need. I told them what I wanted and didn’t ask them to tell me what I need. Have you seen that ad on TV where the software sales guy answers 500 to everything? Seriously it’s like one is telling me I need to decorate and another telling me we have damp and another telling me it’s a tear down ……it’s driving me mad, can you call me on…. 

Scoping application security reviews and disputes that arise from not having scoped application security reviews well are common so I have decided to write the Art of Scoping Application Security Reviews covering everything from “dirty little secrets” of consulting practices  to what to look for when thinking about a code review. This post will be split into several parts (I am planning one a day but you know how it is) and is written for both corporate people working with consultants and vice versa. I think much of the content will be useful to internal teams as well.

The planned parts in this mini-series are;

· The Art of Scoping Application Security Reviews (Part 1) - The Business

· The Art of Scoping Application Security Reviews (Part 2) - The Types of Testing

· The Art of Scoping Application Security Reviews (Part 3) - Threat Modeling and Architecture Reviews

· The Art of Scoping Application Security Reviews (Part 4) - Penetration Testing

· The Art of Scoping Application Security Reviews (Part 5) - Code Reviews

FYI I am planning on a mini-series of posts about building SDL’s or Secure Development Lifecycles in a week or so. I will cover reviewing SDL’s or the big picture of software security programs at that time. Stay tuned for that !

Part 1 - The Business

Why start a series of articles about scoping web application penetration testing with an mini-exposé about how the information security consulting business works? Quite simply I used to assume that this was common knowledge but from recent conversations with some people it seems it not. There is certainly nothing scandalous, new,  underhand or immoral being described here. It’s just how business works and its worth understanding! I’ll cover;

· Fixed Price or Time and Material?

· Rate Cards

· Tools or People?

· Getting Real About Testing

· Bling Bling or Bang Bang?

· Chefs, Restaurants and Food Chains

Fixed Price or Time and Materials?

Nothing polarizes people who run consulting businesses and those who consume services more than religion about fixed price or time and material (T&M) discussions.

With a fixed price contract “Company A” will agree to do the work (usually defined by a statement of work) for a fixed price.

With a time and materials (T&M) contract “Company A” will agree to do the work (usually defined by a statement of work) for an hourly or daily rate.

The benefit to the customer of fixed price is clear; he / she knows exactly how much he/ she will be paying upfront and can budget, raise a PO and plan accordingly. On the surface fixed price work is attractive to corporate customers but in reality they can often find themselves short-changed unless they know the game or make provisions in the contract (see below).

For the consultant fixed price work is a little more complicated. The consultant is essentially making a bet that he/she can estimate the amount of work to be done reasonably accurately. Consultants are then making a bet that they can do the work faster than the time the client thinks it will take them. Their effective hourly rate will be high if they win the bet. What happens if they lose their bet? Not all but a LOT of companies will simply reduce the scope of work behind the scenes so they still get their standard hourly rate. It’s a common practice.

T&M has a bad reputation among some people. You often hear “once they get in they find more and more things and the bills just grow” or “I don’t see why I should pay for their research”. The reality for consultants is that T&M is a life-saver when no one knows what the real scope of the engagement is and most people who buy testing services will acknowledge (even if it’s only in private) that that occurs frequently.

The key to fixed price and time and materials work for both parties is in defining an accurate statement of work. A good statement of work will describe in unambiguous language exactly what is expected of both parties. It is the agreement upfront of what will (and what won’t) be done. If you don’t have an accurate and detailed statement of work then in general expect trouble.

Note: Buyer beware that a proposal is not a statement of work. A proposal is generally a set of marketing claims to encourage you to buy.

Note: Don’t confuse “an estimated price of” with a fixed price contract. Some companies will use this phrase so their proposal looks like a fixed price engagement but the terms and conditions will actually be time and materials.

So what is the best and where should we use each one? Like many things in life there is no one right answer. Assuming you have a good statement of work in place, fixed price is generally considered better for work where the scope is well defined and understood and time and materials for work where the scope cannot be defined upfront or maybe variable. Those astute readers will now being saying to themselves “hang on, this is security testing we are talking about and no one can predict what they might find before doing it so everything should be T&M right?”. In my experience I have found there is a good balance to be found between the two, that is to say a blend of fixed price and T&M works best. Use a fixed price contract for the things that are predictable with a T&M extension to deal with the inevitable issues that will occur provides predictability with flexibility. You can agree when and how the T&M part can be invoked and even limit the financial exposure of T&M work with appropriate language.

Lastly but not least when it comes to time and materials beware of the daily rate and actual hourly rate syndrome. Some companies will quote a daily rate with fine print of billing hourly. The daily rate maybe $2,000 USD with an hourly rate of $250 USD. Of course most firms will then work 10 hour days which means the effective daily rate is actually $2,500 USD.  I would actually encourage all consulting firms to bill by the hour as some clients can have very “high expectations” the amount of hours in a day! This provides a safety net. A good firm that wasn’t taken for granted will likely make some concessions in the billing of extra hours to win follow-up work.

Rate Cards

Most consulting companies have rate cards. They may not appear on menus and sales people usually don’t want to talk about them but most consulting companies have rate cards. The theory goes that the more work you do, the better the client you become and the better rate you get but rate cards are often a double edged sword. Many companies try to use the rate card approach to negotiate down the cost work without making any commitment to volume. This is just business and standard procurement practice from large companies. From a consultants perspective its often a sign that the relationship will be built around cost and not skill. Much like fixed price contracts above what often happens behind the scenes is a shell game. The consulting practice will use cheaper less qualified staff, spend less time looking for specific issues and make their margin on the back end. This is just business. It also goes without saying that most consultants have had the carrot dangled in front of them too many times with promises of millions of dollars if they just start this gig on Monday or drop their price to X. Much like fixed price contracts being tied to statements of work, rate cards should be tied to commitments to volume (incentive for the consultant) and quality of staff / quality of work (commitment to the customer) to become an effective tool on both sides.

Tools and People - I think at this point in the maturity of the application security industry most people have finally widely acknowledged that tools can be useful but are not a panacea. Equally people don’t scale well. Put another way a tool alone can’t do the job and a person alone will likely be too expensive. It’s a combination of the two that will likely provide the best balance between cost and assurance. Viewing the technical problem space (i.e. ignoring threat modeling, architecture review etc) with a more granular lens, certain types of tools are better at finding things than others. A web application scanner has no chance of analyzing code access security in the .NET framework for instance or if adequate logging is in place. It’s well worth the effort of understanding the parts of the scope where automated tools be used and which parts will require humans. I like to think of two classes of issues, bugs and flaws. Flaws are those architectural types things, “flaws by bad design” (poor password reset scheme or logic that allows you to bypass security controls) and bugs as implementation problems (cross site scripting). While far from a hard and fast rule tools are not good at finding flaws but can perform OK finding bugs. As a customer you want to know that the consultants are not wasting your time and money where the cost / benefit of using tools may make sense and as a consultant you want to prove that point to the customer. Application tests should use tools where they make sense and humans where they make sense. A good business will understand this and suggest a whole solution.

We have all seen bait and switch used on TV reality shows or in crime dramas. Sadly it’s often used in the security industry. The guru guy with “a name” is wheeled out, speaks to the customers, impresses them with his / her experience, tells them about his book or his latest Black Hat appearance and wins the work. The fresh faced grads then get fed some more pizza and do the work. The quality of security testing is directly related to the quality of the people doing the work. I would strongly encourage all customers to know the actual people doing the work and if appropriate name them in statements of work. At the same time I think it is worth questioning if you really want “a name” doing the work. Some people that speak at conferences about software security can talk the talk but not walk the walk. In my experience many of the really talented testers don’t like publicity and so operate behind the scenes. It is of course hard to find these folks but it’s a small industry and personal recommendations should go a long way. The key thing to remember is people do testing and not companies. Hire people.

Getting Real About Testing

Suit: “What are you actually going to test for?”

Geek: “Everything and more.”

Of course we all know about romances and this is just romantic. How about this?

Suit: “What are you actually going to test for?”

Geek: “Don’t worry about it, we know what we are doing. We do it for a living”

Information security is neither art nor science. You can read books about code as poems, psychological theories about why people architect things the way they do and hard fast statistics about relationships of code complexity and security bugs. Anyone who has does security testing for a living (and understands it) knows that you will never be able to test for everything because you don’t know what everything is. A good tester will probably be logical and systematic (think neat and ordered) but have flair and a rough edge or two (think well dressed rock star). You want the same in the security testing world. People who maintain check lists and think methodically AND people who can think outside of the box and not follow a script. If you are ever shown a list of “these are the tools we run” or “these are the XSS tags we check for” run a mile. Consider my restaurant analogy below. You probably want a chef below and not a burger flipper!

Bling Bling and Bang Bang?

Most testing companies will spend 25% of their time writing pretty reports. The organized ones have Word macros to insert standard text which can then be customized, the unorganized ones will cut and paste from a previous clients report. We have all seen it. Most consultants have done it and those that claim not to are probably liars. I am just being honest. Reporting is a strange topic. Technical consultants are generally not good authors and hate writing reports. Most companies don’t have time to read long reports or care about the lyrical waxings of a geek and usually mock at the “strategic recommendations for better business”. I strongly encourage both sides to think about better ways of reporting such as using defect tracking systems. Testers will be happier and companies can get around 15% more work for the same cost! Get more bang and less bling.

Chefs, Restaurants and Food Chains

For a guy with a limited food repertoire its amazing I consider myself a foodie and wino. I think I may start video blogging home cooked food and wine tasting with security conversations or something…..in the meantime consider this analogy.

Chefs – a good chef is an artist and a craftsman . After years of training and practice, he can take the finest ingredients and create a masterpiece from his knowledge. It will not be economical but will be the best you can get. It will not be appreciated by all but for those with a fine palette and good taste its nirvana. Even chefs had good and bad days but they live and die by their reputation and operate from instinct.

Restaurants - commoditize dishes so they can be repeated. They vary in quality and appeal to a wider audience. They use menus to control quality. Good ones allow the cooks to improvise and restaurants range from great to poor.

Food chains – produce mass food as cheaply as possible. They operate using formulas and scripts and usually service up highly processed junk. It’s not about individuals but about the company and not about quality but about consistency. (O’Naturals and others like you are obviously exempt from this mass generalization)

 When scoping an application security test you have to decide if you want a particular dish cooked by a particular chef, just want to eat at a particular restaurant (good enough for the majority) or are happy with junk food from a chain.

With all that business stuff over tomorrow I will move on to talking about the types of testing. I will discuss the advantages, disadvantages and when to consider each type. I’ll include;

Design and Architecture Reviews

Threat Modeling

Code Review

Penetration Testing

Later in the week well get to discussing how to size a code review, tools you can use to determine the size of a web application penetration test and other practical matters.