Missed Critical Findings in Penetration Tests

Having talked to many companies, it seems the biggest disappointment when hiring penetration testing consulting firms is the missing of critical findings.  This seems to be a widespread problem.  Many of these companies were hoping to improve their security posture by hiring big security consulting names in the field hoping that they would do a good job.  After all, it's a brand name and well marketed.  It is to their disappointment when a hacker weeks later breaks system using a obvious methodology totally missed by the big name consulting firms.

I have often reviewed pen test reports by the big brand name security firms and have often found them to be quite lacking.  Some of them are even inaccurate.

To understand why this happens, one needs to take a look at the hiring process of many of these consulting firms.  They often hire recent college graduates with little or no experience in security, train them up really quick, and start billing out pentesting at $250/hr using these recent college grads.  The main thing these consulting firms look for is willingness to learn security, and the willingness to work overtime.  Since these companies are highly greedy, they pay these college grads low salaries, make them travel 50% or more, and put a lot of pressure and tight deadlines on them to produce high quality with the shortest time possible.  Often these college grads are desperate to get into the security field and willing to submit themselves to such torture in order to break into a new future.  They hope that in the long term, they payoff will be there.

The sales team bills these inexperienced people out at top rates, charging $10,000-$15,000 for a security assessment.  What they don't tell you is that for 10,000, it's only a week's worth of work, if that.  Often times these grads are made to do 2-3 of these full engagements in one week.  The effective bill rate ends up being $500-$750/hr.  The grad gets $30-$40/hr depending on experience.  Their highly experienced senior consultants get $50-$60/hr out of this.  The client company thinking that they pay good money for penetration test by a big brand name, thinks they will get good results.  The college grad is forced to produce results really fast, not really understanding the application, and really not even knowing security that well to begin with. The grad turns in his/her best work, but it often falls short of true quality.

Understand that the security field is a fast growing market with demand over exceeding supply.  There aren't that many talented security people and the greedy companies hire newbies in order to capitalize on an exploding market to make the most profit that they can.

In order to avoid being taken advantage of by these consulting firms who are exploiting the market, one must look at the person doing the penetration test itself.  What is the background?  What is the experience level?  Demand the resume of the consultant doing the penetration test.  Interview them.  Demand an adequate amount of time be spent on the penetration test before signing any agreement.  Don't just look at the background of the consulting firm.  They for sure do have a few talented people in the organization, but they are spread far and few in between the newbies and inexperienced folks.  Look for companies you know and trust and referred by people you know well.  Having an expert in the field do a penetration test is way better than hiring a consulting firm.