I hope you had a chance to listen in to the Industrial Defender sponsored webinar on Tuesday. If not click on this link to hear Patrick Miller, Eric Byres, Andrew Ginter, Mark Zanotti and myself opine on the subject.

I think the webinar had a great overview on Stuxnet from Patrick Miller and some additional detail from the other panelists. And I think we covered the larger implications of this type of attack well because it illustrates so many concerns in a way we couldn’t before without a data point. That said, I’m left with a feeling that there are so many important questions that remain unanswered such as:

• Have any control systems been impacted? How many WinCC applications sent out their info before Stuxnet was identified? Siemens says two instances and none in production, but they can’t know anything that is not reported to them.
• What was the motive of the attacker? Prove it could be done? Disgruntled Siemens support person or partner – - I’m surprised this has not been discussed more? State sponsor cyberforce – - after all Iran was hit hardest?
• How directed was this attack? Was there a specific target or targets that the attacker was trying to exploit?
• Was this all? This is related to the APT drum I’ve been pounding. Was this the first phase? What else is lurking on a compromised network?

We may never know any of these and other answers.

On Tuesday I’ll be participating in a panel discussion / webinar on the Stuxnet worm. Industrial Defender is organizing it, and there is still time to register. I’ll post a replay link when it is available as well.

It should be an interesting discussion with Patrick Miller moderating and Eric Byres, Andrew Ginter, myself and Mark Zanotti on the panel.

Pay attention to the P in Advanced Persistent Threat [APT]. Most of the attention paid to the trojan with a payload targeting Siemens control system applications has been on the Advanced nature of this malware. And that attention is warranted because there has not been a public example of malware targeting control systems prior to this.

But now that we have had a few days to chew over that, I’d like to pose the question of how do we know the threat has been removed from any targeted control systems and organizations? There has been much buzz on APT recently, including in the control system community after S4. A lot of the buzz has been to sell products and services, but let’s look at that persistent nature that really is the distinguishing feature of APT.

A company finds that it has been the victim of the attack. They investigate; find the exploit; figure out how to clean it out; and hopefully find a way to prevent it from repeating. However, a few days, weeks, months later a different exploit is found. And this happens again and again. There are a wide variety of exploits, in different systems, that wake up at different times, because the attacker had a strong desire to maintain a presence on your network once they had breached it. Maybe the attacker plants a logic bomb in case they are unable to contact one or more of their connections for a period of time.

This attacker directed their attacks on a specific target because that target had something of great interest or value to the attacker.

Based on the available information, the trojan was passed by USB which can indicate it was a directed attack, and the trojan was gathering project information. Perhaps this information gathering is only the first stage of an attack. If your control system was compromised, how do you determine if you have eradicated the attacker’s presence and capabilities on your system?

Cleaning out and preventing the reappearance of the trojan is necessary but maybe not sufficient. I would be very worried where else the attacker is lurking in the system. We know that many control systems today have little patching, minimal security configuration, shared and default user accounts, … So it is likely that the attacker has compromised multiple systems in multiple ways if they wanted persistence.

PS – The Siemens press release on the trojan conveniently puts all of the blame on Microsoft and does not mention their password issue. This is disappointing, but all too common reaction the first time the control system group gets hit with an issue like this. Better to just be straightforward, take the hit, solve the problem and move ahead. It gives all involved some sense of comfort that a strong security group is on the case.

First things first, we’ve been given the application we’re going to access, we’ve built up our testing environment, usually a virtual machine or some sort, and we’re ready to get going.  The application in question serves as a SCADA server, historian, and can serve HMI displays through a native client or a web interface.  Using tcpview from the sysinternals toolset we can see what ports the application is listening on and usually get a good idea about whats going on.  In this assessment we see 80/tcp, 81/tcp, 444/tcp, 445/tcp, and 5481/tcp.  Having setup the application I already have a good idea of what these ports are, but even without having done that you’d have an idea that the first four probably have something to do with web traffic, and the other is probably a proprietary service of some sort.  Turns out 81 and 445 are web interfaces, 80 and 443 are xml (I don’t know why they did it this way instead of the other way around but I’ll let that one go).

A management interface is usually a good place to start, it gives you more of a feel for the things that the application is doing, and since it’s usually not given as much attention from the developers theres often some easy wins there.  Starting up Burpsuite, we’ll use it as a proxy while we browse through the website to have a record of the sites visited, see some injection points, and be able to replay anything we find interesting.

Authentication:

We find some interesting things here.  First, the user is allowed to login over non SSL connections, but thats something the administrator can configure, lets dig a little deeper.  Looking at the way that authentication is handled, once we login with a valid username/password the application sets a cookie in the browser to a randomly generated GUID.  That’s good, they probably aren’t guessable, and when we try to reuse the same cookie to authenticate from another system we aren’t allowed, so that means its tied to a specific IP address.  All that would be reasonably good security, except that one of the pages on the website allows unauthenticated users to view all the valid cookies and the IP addresses that they’re associated with.

Having finished the first round of our manual assessment, its time to let burpsuite’s scanner loose on our target.  It will spider all the pages on the website, and then will inject various payloads into each of the parameters we found during the spidering.  And we find quite a bit of…

Improper Sanitation of Parameters:

The logon page takes in a redirection parameter that is improperly sanitized.  This parameter can not only be used for redirecting, but also header injection since it allows newline characters to be inserted.  This can be leverage to preform just about any cross site scripting attack or redirect from the secure to the insecure website for some man-in-the-middle badness, which is especially important in the case of the web based HMIs, but we’ll get to more issues with those in one of the upcoming posts.

We have tried to find ways to give loyal blog readers a view into how Application Assessments are done and how bad the situation is with many control system applications.

Recently Daniel spent a couple of days black box testing a widely used control system application for an in-house project, and as we were writing up the vulnerability notes we discussed the process and the common findings. After some thought and review, we decided we could sanitize the information and provide information on the type of testing he typically performs on a few of the application interfaces and the common security problems identified in this very short testing.

This blog is an intro to a multi-part blog series that Daniel will run the next two weeks. This is not theoretical or worst case. This is from a real control system application and typical. One thing that was very clear was the “design decisions” did not consider or discounted the fact the application would be attacked.

Terminology Note: Digital Bond differentiates between a Security Assessment of a SCADA or DCS and an Application Assessment in two ways. One, our Security Assessments are typically performed for owner/operators, and Application Assessments are performed for vendors. And two, our Security Assessment is evaluating the SCADA or DCS against good practice and looking for know vulnerabilities. Our Application Assessment is looking for new, as yet undiscovered vulnerabilities. In the best case our, or someone else’s, Application Assessment service would be worked into a vendor’s security development lifecycle.

NERC issued an advisory on Rockwell Automation PLC/PAC vulnerabilities. It is odd in many ways.

1. There is no new information. This is all old news.

2. So many field devices used in this electric sector have these same or equally important security deficiencies. Are we going to see NERC Advisories on every brand and model? If not, why the pick on RA?

3. It is far from complete. For example, they don’t mention that an attacker can load rogue firmware on the Ethernet card because upload is not authenticated. This is similar to the Boreas vulnerability, covered in an S4 paper by Daniel Peck, and proven in our lab loading both innocuous and nasty firmware on Rockwell and Koyo/DL products. This was not hard, but many without the technical background didn’t believe it until they saw it. The preponderance of field devices have this problem.

4. And perhaps the worse, some of the mitigation recommendations are just wrong. They suggest using FactoryTalk, but this means no password is used on the PLC/PAC so an attacker does not even need to intercept the password. See our SCADApedia note on this. Other mitigation is simply restrict access – - don’t let the bad guys get to it.

This seems very unfair to Rockwell Automation. Perhaps a more generic advisory about typical vulnerabilities in field devices would have been better. RA has some talent on their security team, and I believe the message is starting to get to the exec’s there and elsewhere. Things don’t move as fast as we would like to see, but there should be some good options for security in field devices in the next year or two.

It’s RSA Conference time so companies have reports and studies to release. One that I actually found interesting is Veracode’s State of Software Security. The data comes from assessment of “billions of lines of codes and thousands of applications.” It provides some good data points and observations on the state of things.

I’ve gotten to where I read any InfoSec literature through a control systems lens. I’m always asking, “how does this apply or not apply to the control system applications and hardware we work with?” In this case, the relevance is easy to find. The distribution of vulnerabilities by language, for example, is an interesting educational tool for developers regardless if the application you’re writing is balancing a checkbook or opening a breaker.

For those of us that are looking at security every day, there are few surprises here. What we have to do if find ways to educate people that there is a problem. That’s where reports like this can be useful. Here are Veracode’s key observations:

1.) Most software is indeed very insecure.

2.) Third-party software is a significant percentage of the enterprise software infrastructure, and third-party components are a significant percentage of most applications.

3.) Open source projects have comparable security, faster remediation times, and fewer Potential Backdoors than Commercial or Outsourced software.

4.) A significant amount of Commercial and Open Source software is written in C/C++ making it disproportionately susceptible to vulnerabilities that allow attackers to gain control of systems.

5.) The pervasiveness of easily remedied vulnerabilities indicates a lack of developer education on secure coding.

6.) Software of all types from Finance and Government sectors was relatively more secure on first submission to Veracode for testing.

7.) Outsourced software is assessed the least, suggesting the absence of contractual security acceptance criteria.

Each of these points is interesting and could be its own blog post in its own right but I want to focus on number five for now. I absolutely believe that more can and should be done for developer education. But the other lens I look through now (thank you, Ross Anderson) is the economic perspective. What is the incentive for developers to do better with secure coding in the control system space? Until customers require it, it’s just not going to happen.

Perhaps the most effective way that customers can require more from vendors is at the time of purchase. Easy win here: the procurement language document has been out for a while now and contains a secure coding section. A while back I conducted an informal survey of control system vendors regarding use of the procurement language in RFP’s. The results are disappointing. One positive outlier was at 40%, but most were closer to the single digit percentages.

I agree with the report — we’ve got education to do, but for our space it’s not just at the developer level. How much longer will it take to for us to have the at least same security expectation levels of our critical infrastructure applications as we do when we order a pizza online?

Digital Bond offers an application assessment service if you want to know what the “state of things” is for your software.

We selected Kris Harms from Mandiant to give next week’s S4 Keynote on the topic of Advanced Persistent Threat [APT]. This week Google and Adobe announce investigations of some more serious than normal attacks. A couple of key excerpts from the Google blog:

In mid-December, we detected a highly sophisticated and targeted attack on our corporate infrastructure originating from China that resulted in the theft of intellectual property from Google. However, it soon became clear that what at first appeared to be solely a security incident–albeit a significant one–was something quite different.

and

Second, we have evidence to suggest that a primary goal of the attackers was accessing the Gmail accounts of Chinese human rights activists.

If a country or organization is willing to target sophisticated attacks to learn info about human rights activists and potentially get commonly used source code, is it a stretch to believe they would apply similar resources and talents to learn how to attack critical infrastructure systems? And importantly maintain that capability – the P in APT.

As a community we need to address the most frequent threats of malware, script kiddies and non-targeted attacks, but we also need to start looking for and preparing for a much more talented and dangerous adversary. The looking for is important because would most control systems know they have been penetrated if the adversary chose not to affect the system yet?

I wanted to wait to hear the reactions to the segment on 60 Minutes before commenting. If you missed it, see it here or read the transcript. Here are a few thoughts on the story.

I was out at EnergySec in Seattle last week, and tweeted on it @digitalbond.

An INL presentation showed that they have found about 325 vulns in the control system assessments they have performed over the last four years. This revived my long held and stated frustration about who gets this information. When INL does a vendor assessment, it is frequently paid for in part or in full by the US Government. Your tax dollars at work.

INL signs an agreement with the vendor being tested that the results will only be shared with the vendor and the sponsoring USG agency. So the vendor has sole authority on what is done with discovered vulnerabilities. Some have chosen to address the vulns, and then provide the full report along with the fixes or corrective actions to their customers under NDA. We know this because they have INL provide the full report directly. Bravo.

However some have chosen to provide the positive excerpts or highlights of the report and remove any vulns or problems they do not intend to fix. Even worse, they can say their system has undergone INL testing giving it some implied certification.

Other vendors have chosen to fix problems in their systems, but not tell the customers about the security problems or corrections – - the dreaded silent fix. Owner/operators using the system often choose not to upgrade to a new version absent a compelling reason. So key security fixes are not factored in the upgrade decision.

Allowing the vendor to have sole authority of how the results are shared may have been necessary when the program started, although even this is debatable, but now the INL test program carries so much weight with potential and existing customers that INL / USG have more negotiating clout. They could require more information sharing with the affected customers. For example, the vendor could be given six months to address all findings or develop compensating controls before the vendor or USG must share the information with affected users.

Is an owner/operator better off knowing about an unfixed vuln, or is it better to keep this information only within a small sphere in the vendor? After all, the more people who know something, the more likely the information will leak out to bad actors. I would argue that an owner/operator needs to know the information. If INL could find a vuln, then others with access to the system could as well. As an owner/operator I want to know the vulns so I can make risk decisions on compensating controls and my use of the system. I have yet to see a vuln that does not have some compensating control, so don’t tell me you are keeping information from me for my own good. Owner/operators are not children.

So why is the information not being shared with affected owner/operators? I really don’t know. In discussions over the years I’m convinced that the INL researchers want it shared. After all, who wants to keep their results from those it could help? I’m also convinced the government wants to share it. The vendors don’t want to cede control over the results, but again the program has so much clout now that some delayed disclosure is possible.

I don’t know why we are stuck in this vendor discretion model, but my best guess is the financial and legal people that run INL are the impediment. Changing the status quo has no legal or financial benefit to the lab, and they can argue that it exposes them to an increased legal and financial risk. With no benefit and even a small risk, why rock the boat? Absent some pressure from the USG, I see no chance for change. Maybe at the next Congressional hearing some panelist can ask why a National Lab is not being required to share known vulnerability information with affected critical asset owners when the work was paid for by the USG. As much as I grouse about Congress getting involved in control system security, it may be a call from the right Senator or House member that is the only thing that can change this model.