Bit9's APT Hacker Break In

We often hear of security companies who provide protection for other companies get breached.  So what's going on here?  How is this possible?  Honestly, I work with a lot of these security vendors and a lot of them don't know security or implement security pretty poorly in their products. This week, let's take a look at Bit9.  From their website, they claim to "continuously monitor and record all activity on servers and endpoints to detect and stop cyber threats".  Yet their break in, according their admission in their blog released yesterday, claims it was a SQL Injection attack that was undetected for at least 7 months.

The attackers, it seems broke in through a SQL injection attack, installed a malicious rootkit, cracked two additional user accounts, and grabbed the signing certificates.  All this activity went on without being detected by their infamous Advanced Persistent Threat Detection security monitoring software and their 24/7 security operations team which monitors everything.  lol.

The only way they found out about this was 7 months later when their customer complained they had malware running in their trusted network.  The malware was digitally signed by Bit9 as "trusted software". 

There are a few questions one must ask regarding this:

1. How did all this activity go unnoticed?  Why did Bit9 not install their own security monitoring software in the DMZ where they hosted the vulnerable machine?  That's a violation of security policy, in itself.  Did they not trust their own software to do the job right? lol

2. Why was an application not security tested before being released to the Internet?  That's a violation of another security policy.  SQL injection is one of the easiest vulnerabilities to find.  A script kiddie could have found this.  A half-decent web scanning tool would have found this. 

3. Access was gained to the server containing the signing certificate.

1. Why are the signing keys available on their Internet facing server?  It should be on an isolated network, if on a network at all.  This is a gross violation of key management.
2. Is Bit9 also claiming now that they don't even run their security monitoring software on the network that contains the keys to their signing certificates?  This is their trade secret and the back bone of their entire business.  If there's one place to run it, it would be here.  It's a gross violation of security policy not to monitor this.  It's more likely that they did run it there but never found the incident.  Bottom line: a lot of security products out on the Internet are fraudulent.

4. Bit9 claimed it was not detected because the system was shut off.  That's the purpose of security monitoring software.  Even if the server had been turned off, all the malicious activity should have been recorded and reported to the Operations Center.  Hello.

5. Bit9 then claims that they discovered the break in shortly after it was turned back online in January 2013.  It wasn't detected before while it was online but now since it is back online, it is detected?  Either they are monitoring the server or they are not monitoring the server.  Which one is it?

6. If Bit9 monitored customer networks 24/7 with a live security operations team, then why was it the customer who had to inform Bit9 that they were infected?  Think about it.  It was running on customer networks for 7 months without detection by their Advanced Threat Indicators (ATI) and  24/7 monitoring Bit9 was supposed to provide.  In the end, it was the customer who had detected it and complained to Bit9.  Bit9 then did an investigation and found malware on additional customer sites.  It was the customer who leaked to the press what really happened because they didn't like the way Bit9 had presented it. You have to wonder what kind of monitoring Bit9 does and more importantly, what value it provides? ;)

All in all, Bit9's account of what happened is inconsistent and highly questionable.

Lessons Learned From This

1.  Companies that do not have security policies, standards, and controls in place are *not* to be trusted with security.  People should refuse to do business with a company that fails to protect itself and fail to detect the security incidents they claim to be an expert at. It seems like an employee could just pop up a Internet facing application on Bit9's DMZ without any kind of protocol or control being enforced by ANYONE.

2.  All applications should be security tested before being released to the Internet, even insignificant ones.  A chain is as strong as its weakest link.  If all the other servers are hardened, but they leave one highly vulnerable server running next to it, then the other servers could be compromised.  That's like locking all doors to the house but leaving the windows open thinking the house is well locked. lol

3. Use security monitoring software that can actually catch incidents.  Have a Pen Tester attempt some nefarious activity and see if it was caught.  And by this, I don't mean run a Nessus scan, or even a port scan. lol.  Anyone can catch those.  Do an APT (Advanced Persistent Threat) break in. 

4. Have security policies, standards, controls as to what is allowed on the DMZ.

5. Have a strong key management policy.

QualysGuard Web Application Scanning v2 Review

Having done numerous penetration tests using various manual and automated tools, today we are focusing on a new tool called QualysGuard Web Application Scanning v2.4.1.  In the process of doing a pentest, we often use a quality automated tool to check for standard issues while we focus on the much more difficult issues of the testing.  As this reduces the time it takes to do a full test, allows us to work more efficiently, and besides who wants to waste time doing monotonous simplistic checking.  In this regard, I have used AppScan quite extensively, and HP WebInspect as well, and both are very good tools for the most part.  They help out on the basic checks quite a bit.

Quite recently, I was introduced to QualysGuard Web Application Scanner (WAS) v2.4.1.  This tool was very simple to use which is true to Qualys name.  Point and click and you are done.  Unfortunately, I found out that it didn't help with the standard checks either.

Problem #1
1. It couldn't even authenticate to basic web forms.  I've used AppScan on hundreds of sites, and not once was there a problem in not being able to authenticate.  A web security tools isn't very useful if it can't get passed the logon screen because that's where most of the application resides.  How is it supposed to check anything if it doesn't get passed the logon screen?  The Qualys product support/product manager's response to this is to use Selenium Scripting.  Unfortunately, the current applications that are being tested only run on Internet Explorer (IE) and Selenium scripting automatic record and playback only works on FireFox.  So one must learn a new scripting language in order to make it work with IE.  This is hardly an easy point and click solution.  Learning a new scripting language is time consuming and error prone.  Other professional web scanners have this feature built in.

Problem #2
2. It does not have a built-in manual explore like other professional tools.  For instance, manual explore is needed to fill in certain forms properly in order to get to the critical screens for testing.  For example, you must fill in a proper social security number to look up the customer and get to the rest of the application.  Qualys WAS does not support this feature.  This web scanner doesn't allow the user to fill in the initial forms with proper data thereby never testing the whole application, which is critical.  The Qualys product support/product manager's response was this is a simple point and click tool, "we don't support nor do we plan to support complex features such as manual explore." 

Problem #3
3. Web service scanner has limited functionality in comparison to other professional tools.  In this day and age, many web applications use web services.  To not support this feature properly is ridiculous.  The Qualys product support/product manager's response, "we only support web service fuzzing at this point."  What about testing authenticated web service calls?  It also doesn't support pre-populated data on web pages not web services other than the logon screen.  This pretty much reduces their web service testing to a dummy tool.  To make this work, you have to use tools like SOAPUI or Burp Suite Pro with scripting/plugins to pre-populate data, manual explore, and sequence test steps.

Problem #4
4. Lack of details provided by Qualys.
a. Most professional tools have an audit log that shows exactly what tests were performed and how they were performed.  Qualys does not provide an audit log of what tests they did.  We are supposed to guess instead as to what might have actually transpired.  Real reason behind not providing an audit log is more likely along the lines of they don't all the check they are supposed to and even if they did,  it probably wasn't exhaustive testing of say XSS.  Either way, we have no idea whether they did the work they claimed to have or not.  A Big Mystery Here!

b. No details provided on the actual request/response when a vulnerability is found.  True to Qualys name of simplicity.  The vulnerability finding is so simplistic and lacking any details as to how it was tested, one wonders how to test whether this finding is a false positive or not.  Well, I guess one is supposed to take Qualys word for it. :)

Problem #5
5. Missed critical session management vulnerabilities.  Qualys missed a critical session management vulnerability that I had to find manually that AppScan would have found.  The Qualys product support/product manager's response, "we are putting in a fix for this soon."

All in all, QualyGuard Web Application Scanner (WAS) v2 is lacking quite a bit in terms of quality and details.  Do you want to risk the security of your enterprise by relying on a product like this?  Currently, the product is premature and should not considered to be a proper product to used for PCI approved Web Scanning.  In fact, it should not even be PCI approved until it matures quite a bit.  Qualys needs to understand how a true web application scanner works before releasing a premature product to cash in on a exploding market.  

Update 05/20/2013
A coworker has confirmed the below:
Problem #1. With the latest patch, Selenium Scripting has been successfully used to log into basic web forms.

Problem #2. Selenium Scripting now works for doing manual exploration.

Missed Critical Findings in Penetration Tests

Having talked to many companies, it seems the biggest disappointment when hiring penetration testing consulting firms is the missing of critical findings.  This seems to be a widespread problem.  Many of these companies were hoping to improve their security posture by hiring big security consulting names in the field hoping that they would do a good job.  After all, it's a brand name and well marketed.  It is to their disappointment when a hacker weeks later breaks system using a obvious methodology totally missed by the big name consulting firms.

I have often reviewed pen test reports by the big brand name security firms and have often found them to be quite lacking.  Some of them are even inaccurate.

To understand why this happens, one needs to take a look at the hiring process of many of these consulting firms.  They often hire recent college graduates with little or no experience in security, train them up really quick, and start billing out pentesting at $250/hr using these recent college grads.  The main thing these consulting firms look for is willingness to learn security, and the willingness to work overtime.  Since these companies are highly greedy, they pay these college grads low salaries, make them travel 50% or more, and put a lot of pressure and tight deadlines on them to produce high quality with the shortest time possible.  Often these college grads are desperate to get into the security field and willing to submit themselves to such torture in order to break into a new future.  They hope that in the long term, they payoff will be there.

The sales team bills these inexperienced people out at top rates, charging $10,000-$15,000 for a security assessment.  What they don't tell you is that for 10,000, it's only a week's worth of work, if that.  Often times these grads are made to do 2-3 of these full engagements in one week.  The effective bill rate ends up being $500-$750/hr.  The grad gets $30-$40/hr depending on experience.  Their highly experienced senior consultants get $50-$60/hr out of this.  The client company thinking that they pay good money for penetration test by a big brand name, thinks they will get good results.  The college grad is forced to produce results really fast, not really understanding the application, and really not even knowing security that well to begin with. The grad turns in his/her best work, but it often falls short of true quality.

Understand that the security field is a fast growing market with demand over exceeding supply.  There aren't that many talented security people and the greedy companies hire newbies in order to capitalize on an exploding market to make the most profit that they can.

In order to avoid being taken advantage of by these consulting firms who are exploiting the market, one must look at the person doing the penetration test itself.  What is the background?  What is the experience level?  Demand the resume of the consultant doing the penetration test.  Interview them.  Demand an adequate amount of time be spent on the penetration test before signing any agreement.  Don't just look at the background of the consulting firm.  They for sure do have a few talented people in the organization, but they are spread far and few in between the newbies and inexperienced folks.  Look for companies you know and trust and referred by people you know well.  Having an expert in the field do a penetration test is way better than hiring a consulting firm.