Top 10 Tips For Hiring Web Application Pen Testers
This post was updated on March 8th based on feedback and comments.
Thomas Ptacek’s post about how to choose a penetration tester made me chuckle with his phrase ‘Web Pesting”. He makes some excellent points. Chris Eng originally posted an equally excellent blog. Having ran a sizeable team doing this work at Foundstone I thought I would chip in with my 2 Euro’s worth, specifically on web app pen testers. These tips are based on my experiences and observations of interviewing, hiring and managing these folks as well as being a consumer in previous jobs. I also saw a lot of these folks come and go through the halls at OWASP.
1. Demand Individuals - No two people are alike. The skills and attitude of an individual will be the difference between a testing company meeting their contractual obligations and one exceeding your expectations and providing you with exceptional value. Without fail you should interview an individual and ensure that a named individual or individuals will be the people that actually do the testing. Many consulting companies pull the “bait and switch” and many of the big four use graduates out of colleague. This is understandable economics given the competitive nature of this market but don’t fall for it.
2. Match Individuals to your Application - Web technology is pretty diverse and a thorough underrstanding of a specific technology will get you the best bang for the buck. If you want to test an ASP.NET app, make sure the person really knows ASP.NET under the hood (how postbacks work, CAS, pagestate etc.). A good ASP.NET person instantly knows where the hot spots are. Same for Java, RoR etc. I once was given a guy from IBM who wanted to waste my money and test for overflows in a WebSphere app. Did he know something I didn’t? Not this guy!
3. Road Test - Don’t be afraid to road test the individuals if you are in any way un-sure. I started doing this at interviews and its interesting to see people with supposedly have years of experience miss obvious issues that weren’t on their pervious jobs “call center script”. You can use modified versions of the Hacme Series or OWASP Site Generator, a project I first sponsored, to highlight the effectiveness of people and tools. This doesn’t have to be a laborious process; “here take 15 minutes and this web app of my choosing and show me what you can do. I am not expecting you to find everything, I just want to see how you approach it”. Never use a canned app or let the company demo their own for obvious reasons.
4. Hire People Who Can Code - It’s incredible that a large portion of web app pen testers don’t code. While it could be argued that a large part of the trade is HTTP shenanigans, unless the person understands how an application is built they won’t really understand how to break it beyond the text book. Ensure the individual can at least write code or make them write a small web app pen testing utility on the fly if they have no demonstrable creds. If they can’t, don’t hire them.
5. Be Prepared to Pay for Quality - You can certainly find cheap testers if you want them (and that violate the advice offered on this blog post). Good testers are in demand and can command a good salary. The basic economics means the company will have to charge you the going rate. This is currently anywhere from $200 to $300 USD per hour. Don’t get fooled for the “It’s $10K fixed fee and takes about a week” line. $10K for 5 half days is not good value.
6. See the Methodology - Without getting into a debate about wether you really need a methodology and if its really secret sauce, demand to see the methodology. If they are well organized the company will have documented the things that they will check for, how they will do it. Ignore form, focus on content. A good methodology will have enough freedom for a good tester to follow his nose and find blood as well as making sure they have covered the broad spectrum of potential vulnerabilities. Don’t fall for the “its our secret sauce” or “our guys are so good we don’t need to document it”. Hogwash!
7. Look at Their Toolbox - Ask to take a look at their toolbox. If their proposal mentions dated or rudimentary tools beware. Don’t fall for the “we use proprietary in-house tools” unless you can see them. Would you let a unqualified medical person inject you with a liquid you don’t really know about? It could be water or poison! If they use automated scanners get the raw reports generated from these tools on a data CD. This is not what you are paying for and you want to be able to see the delta. I always recommend hiring people who write tools and not those that simply run them.
8. Ignore the Bling, Focus on the Zing - You want to see good results and not a “bling bling” report. If you get one for free, all the better but don’t pay for it and don’t be fooled by glossy charts. Don’t accept sloppy reports either of course. Consider having the testers dropping issues into your defect tracking system as opposed to report writing. This is often faster and a more effective use of everyone’s time. Agree up front on what you are interested in and how the testers will report issues. Add points if they make solid recommendations for remediation and not just throw up issues. Ask to see a sample report. If they make excuses or tell you it’s out of date don’t use them.
9. Fixed Fee Means Fixed Time Period! - People do testing for profit. If you constrain the cost you will constrain the amount of time someone spends on testing wether the sales guy tells you that or not. My suggestion is build a contract with a fixed fee and optional day rates that can automatically kick in on the mutual agreement of both parties. Encourage the testers to ask for more time for potentially juicy issues and have them explain why they think its prudent.
10. Loose Lips Sink Ships But Get References - I know it goes without saying but sales people are sales people. Get references but don’t trust what sales people tell you. Verify what they say is true. If they ever discuss a “hack” and mention or hint at a company without providing a name and number to call in the same breath run a mile. It will be your issues they are airing in public at their next roadkill session.
I could go on but to make this a manageable read I think these 10 points will stand you in good stead.
10.5 - Word of Mouth is a Great Criteria - The following guys have all worked for me in the past and are absolute rockstars (no order of preference).
Eric Hietzman - Mandiant
Aaron Higbee - Intrepidus Group
Mike Andrews - Foundstone
We are also doing a small amount of select testing to boot strap our software company so of course feel free to drop me a line!
I’ll post “Top Ten Tips for Hiring Code Reviewers” is there is a demand.
March 7, 2007 at 8:51 pm
Thanks for the notable mention!
As a non-coder who learned WAPT,… I’ll share with you a funny little thing that happened to me last year. I was testing a web app that seemed “strange” to me. It was making tons of requests and I didn’t know why. It was a tad annoying to test, but I got through it and found some major vulns.
A few months later, while reading up on AJAX, I realized that the app I tested, was AJAX. I’ve often wondered if I’d be a better WAPT if I had a coding background. I don’t think I would. Just because you’re a coder, doesn’t mean your mind is wired to be a good breaker. Some people are natural breakers no matter what it is. I bet if I was a coder my recommendations would be better. Security doesnt excite me as much as insecurity does.
March 7, 2007 at 9:38 pm
Thanks for the shout out.
I like the ideas in the post and can add a couple.
- Rotate consultants: don’t have the same consultant look at the same app multiple times. Two consultants from the same vendor seems to work out, but getting a fresh pair of eyes on the problem seems advantageous.
- I’d question the rule #7: Look at their Toolbox, especially as it pertains to application penetration testing. I’ve done pen test against applications supported by three different web app vulnerability products, and generally have found that the automated tools are a poor crutch compared to the results from the manual pen test. If anything, I would say “don’t go with a vendor that just runs an automated scanner and flips the results.” Make sure the tools are de-emphasized rather than the focus of the assessment or the report.
- Also, I’m in the same boat as Higbee when it comes to the necessity for a development background. The commercial development experience I have isn’t pertinent to web app pen testing, and I’ve never contributed to any open source code project. I do get web app security inside and have an academic computer science background, but I think there is merit to Higbee’s point that some people are great at (and enthusiastic about) finding security bugs. On the other hand, coders probably provide more pertinent and applicable recommendations than the high-level recommendations you might get from a non-coder, but even that applies more to junior testers than senior ones.
Anyway, great article! Mark’s clearly a thought leader in the space and seems to excel at organizing thoughts into these great top 10 lists that have a lot of industry impact. Keep up the good work!
/eh
March 7, 2007 at 9:58 pm
[...] Mark Curphey - SecurityBuddha.com Security Enlightenment - Mark Curphey « Top 10 Tips For Hiring Web Application Pen Testers [...]
March 8, 2007 at 12:05 am
Glad to see this topic getting some good discussion since I first blogged about it. I have to agree with both Aaron and Eric on #4. In fact, the best pen testers I’ve had the pleasure of working with at @stake didn’t come from development backgrounds. Some haven’t even had CS degrees! It doesn’t matter. Understanding how web apps work and having creativity and determination ends up being more important. Otherwise, great points, and I’m looking forward to the Top 10 for hiring code reviewers!
March 8, 2007 at 12:28 am
Yeah, let me add to the thanks for the heads up
I’d agree with Eric’s comment that tools aren’t that important. You obviously want to use some tools to make things easier (SSLDigger, Nikto, etc), but as the saying goes, “Tools do not make the man”. What I would ask is what the consultants *do* use tools for, and what that *dont*. If they can’t give a good explanation of why they use a certain tool, why it’s better to use a tool or manual effort in particular instances, how the tool works/how they might dev the tool (or even make it better), I’d walk - the tool is clearly a crutch in those cases.
I’ll slightly disagree with Aaron and Eric though with the development background issue. I dont think *commercial* dev is necessary (that’s a different skillset and more to do with teamwork/working tospecs/syncing code, than “knowing what the code does” IMO), but some dev experience is very necessary IMO - people that have an understanding of code, and how a system is probably put together is much more likely to know where to start looking for problems, or even more importantly to back off when things look like dead ends. As much as Aaron and Eric say that they are non-coders, I know that they both have a good *understanding* of code, and that’s what’s important. Saying this though, Eric is spot on in that being a dev helps recommending fixes, but to be a successful tester you really do have to have that (evil) mindset in wanting to break stuff
Just one more follow-up on a couple of Mark’s points about methodologies - there is *no* reason why a company won’t show you their methodology. If they don’t they are most likely going to be doing uncoordinated testing. Sometimes this works great, other times too much time will be spend chasing non-bugs - it’s very hit-or-miss. Methodologies keep you focused, and *they are not secret sauce - especially in the WAPT world*. The company may have specific *testing strategies* that they want to keep close to their chest, but there’s so much out there now for web security testing, there’s no excuse for not having one (and no, the OWASP Top Ten *is not* a methodology ;))
I’d like to add one of my own top tips in hiring WAPT testers - ask them about the latest vulnerabilities out there / attack vectors. If they can’t talk to you about the recent Google desktop hack, what XSRF is, how HTTP response splitting is (or DNS pinning for that matter), and the dangers of them, then also walk - it means that they are not tracking the industry and you are getting tests that are at least a year out of date.
But anyway, great article, and thanks for the props Mark
March 8, 2007 at 8:20 am
A fool with a tool is still a fool!
I will be updating this post in a few minutes. Commercial coding experience was the wrong phrase. I firmly believe that being able to code is essential.
March 8, 2007 at 10:52 am
Our industry is going through a weird period at the moment, with the current batch of security people not really knowing whats going on (except a few)
I remember doing interviews at ABN for good pentesters and you could say i’m biased towards web apps, but I expected talent to come through the door and all i got was the “hacking exposed phenomenon ©”. These were firewall/network engineers who heard that security was the place to be and purchased all the books and kaboom, I is a pentester yeah?
I also work with a strict methodology and anyone who cannot fathom this isn’t right for the job, period
Thankfully my new industry is full of bitches/wankers/ego’s bigger than Dark Tagents and stylists who make Defcon look like a summer camp :p
March 8, 2007 at 3:09 pm
Good update. While I don’t call my self a “coder” I always find the need to script up custom tools for testing custom apps. curl, nc, sh, sed, awk, openssl, stunnel are many of my go-to tools. A good tester DOES need to know how to create loops, etc…
-higB
March 8, 2007 at 3:12 pm
Yeah that was the point I was trying to get across (badly). You and Hietzman and Andrews are masters at that.
March 8, 2007 at 6:29 pm
[...] in the past and contemplating what I will do better next time that leads me to this post. My Top Ten Tips for Web App Pen Testers and Top Ten Tips for Hiring Security Code Reviewers seems to have been well received and it’s [...]
March 14, 2007 at 12:09 am
[...] Write posts like my Top Ten Tips for Hiring Web Application Pen Testers which have a Long [...]
March 15, 2008 at 11:17 pm
[...] (usually for an "average" size site 3-4 hours of scanning isn’t uncommon). As I said in Mark Curphey’s blog way back, "tools do not make the man" - any pen tester [...]