Typing scada as the search key in a Google news search http://news.google.com reveals that as a whole the industry (vendors, asset owners, and security players) still needs to raise the bar on security awareness and must change its mindset in a couple of key areas.

While I don’t want to become a purveyor of FUD, when vendors promote press releases publicizing how they have just sold umpteen millions of dollars of product X to asset owner Z (such as this one found by a google news search), while being good publicity, in some ways gives away the keys to the kingdom. Any attacker interested in targeting asset owner Z now knows what flavor of control system he needs to research, replicate and exploit.

In another vein of thought, the Wonderware DoS attack alert made Google News and it has some interesting implications. In the past the hardest step for a control system attack against an installation with decent security (e.g. good defense in depth with proper; network segmentations, well configured firewalls, access control etc) was to move from either the corporate or DMZ segments and into the control system LAN(s).

The exchange between the exploit team at Core Technologies and Wonderware (seen here in the timeline ) indicates that the com channel exploited in this vulnerability is generally allowed through the firewall. This becomes particularly interesting when you do a search on the terms “web enabled” or “web server” and SCADA and start counting how many vendors are now extolling their new “world wide web” ready and “web enabled” products.

These products push ease of use as their main feature and include verbiage like “… with SMS reporting and remote control to provide real-time access anytime, anywhere through a standard Web browser.” and “… [productx] modules were designed from the ground up using OPC and ActiveX technology making them fully Web-enabled! With these [productx] Web Solutions, you can view your HMI/SCADA application across your local intranet or the World Wide Web through your Internet Explorer browser. ” in their sales literature.

Though I haven’t done a lot of in depth research into the specifics of these products the very idea of increasing the number of exceptions for firewall rules worries me, especially browser based connectivity as the number of client side browser exploits seems to continuously grow. Control system components allowing connections to thin webservers from anywhere on the outside to inside the control LAN through firewall exceptions, frankly gives me the willies. But not quite as bad as the idea of every HMI operator having a browser on his console that allows him not only to use his browser to control the process as the HMI is web based, but to surf to basically anywhere on the “wild woolly web”.

No, this is not what you think it is. In fact, it is almost the opposite.

Control system applications and components are increasingly being used to monitor critical IT Data Centers. We heard quite a bit about this at the PI Developers Conference, especially related to monitoring the power, environmental and system performance in the Data Center.

A recent report by the Gartner Group shows why Data Centers are looking for monitoring assistance. There are huge $$$ in these large data centers in both the cost of running them and the lost revenue for every second they are down.

I couldn’t let the Wonderware Suitelink vulnerability go by without commenting on it, and even Jason commenting on it below won’t steal my thunder.

First, lets talk about the vulnerability from a technical perspective. It appears that this is a fairly classic example of the program allocating an amount of memory based on a request from the user, in the exploit case this normally comes in the form of a very large amount or a negative amount, and failing whether the allocation was successful before attempting to write to the requested memory.

In this the vulnerability doesn’t seem to be able to be used to run arbitrary code on the system, but being able to crash a system/service at will with a single unauthenticated network packet is still bad news. Lesson to learn is always check return values, it’s a lot like learning to check the expiration date on the milk carton before taking a big gulp.

The more interesting part of this vulnerability for me is the process that Core and Wonderware went through. Over the years I’ve read a lot of advisories, and the “Report Timeline” section of this one had me alternating between shaking my head in disbelief and laughing. Here is a gem to let you see what I mean:

2008-03-03: Core sends proof-of-concept code written in Python.

2008-03-05: Vendor asks for compiler tools required to use the PoC code.

2008-03-05: Core sends a link to http://www.python.org where a Python interpreter can be downloaded.

I’m not going to get into the politics of disclosure, as thats been argued enough that everyone knows how they feel and no one is going to change their mind, but the process that these two went through over the course of 3 months was a little ridiculous. I’m not sure if it was stalling, or just unfamiliarity with dealing with vulnerability information, but processes should be in place at any vendor that cares about better securing its products to take vulnerability reports seriously from day 1. There is an understandable amount of hesitation when dealing with companies that release vulnerability details, but from a clients perspective working with them as best you can and having a patch available when the advisory is released is much better than the alternative.

Sebastian Muniz from Core Security Technologies discovered a denial of service vulnerability in the Wonderware SuiteLink service that was made public today. Here are some links:

Core Security Advisory

National Vulnerability Database

Wonderware Tech Alert (login required)

This SuiteLink vulnerability affects the same version of Wonderware InTouch that had the NetDDE problem. When we presented the NetDDE vulnerability at S4 2008 back in January, we actually listed the SuiteLink service as a potential threat in our threat model. Little did we know that Muniz was actively discovering and researching that very issue.

Some of the same topics we addressed with the NetDDE problem still apply — yes, this is old software but the life cycle for these systems is long and, with a product as widely used as Wonderware InTouch, you know there are many vulnerable systems out there. This was corroborated by the number of S4 attendees who confessed to having InTouch 8.0 installed on their laptops. That said, if an attacker is in a position to exploit the NetDDE or SuiteLink vulnerabilites, in most production environments there has already been a serious breach of network security.

This will likely re-ignite the debate over the disclosure process for control system application vulnerabilities. For the record, this is Digital Bond’s policy.

Stay tuned for a full technical write-up in the SCADApedia Vulnerability Notes.

It looks like the DNS service for a few top level domains will be more secure in the future.  Announcements, by way of Dark Reading, have been made that the .org, .uk, and .arpa will soon be turning on DNSSEC and joining .swe (Sweden), .br (Brazil), and .bg (Bulgaria ).  While DNSSEC doesn’t solve all the security problems with the DNS system (in fairness it was never designed with security in mind, and it shows), it does solve a lot of them.

For those of you who don’t know, DNSSEC is an extension that allows DNS responses to be digitally signed, assuring that they are from an authority, and that they haven’t been tampered with.  As phishing and browser exploitation become bigger and bigger parts of security everyday, old attacks like cache poisoning and DNS spoofing that were once only used for hacker pranks have become a bigger part of very real attacks and vulnerabilities like this Apple Update Vulnerability from last year.

So does this mean we’re going to see DNSSEC on the .com tld soon?  Probably not for quite a while, but its a good sign when larger domains like .org and .uk are making the move.

Aside from the specific issue, the process of securing the DNS system is an interesting case study in relative cost of building in security versus patching in security.  As anyone who’s worked in IT or software development can tell you, the former might cause some temporary pain, but the latter qucikly turns into a nightmare of backwards compatibility, feature creep, and a significantly higher cost.  And that costs only becomes larger the more heavily deployed/entrenched the existing solution is.

Moral of the story is the same as it usually is, either develop/deploy with security in mind or at least be aware of where its lacking and mitigate the threats as much as possible.

I’ve been involved to varying degrees with security standards efforts for way too long now - - almost twenty years. Most recently with the ISA 99 Part 4 effort. For a while I was actively involved in that effort in support of a contract with Wurldtech. When Bryan Singer joined Wurldtech that did not make sense any more for obvious reasons, so at that point it was one of many possible pro bono activities. Since then I have been only minimally involved, lurking in an occasional weekly conference call and looking at some of the documents.

So the question we are asking internally was and is: Is actively contributing to ISA 99 Part 4 or another control system security standard the most efficient use of our pro bono time to move the control system security effort forward? This question came up again with the NERC call for technical experts to help with CIP revisions.

Another question is whether a consensus standard that passes a vote is a worthy security document. Is it a representation of good security practice or a least common denominator? I have written before about my angst with Insecure by Default votes. Most involved with NERC CIP will tell you that many requirements were reduced in rigor so the standard would pass rather than because it was what the drafting committee agreed was best.

I admit that we have a few unique advantages at Digital Bond in this decision. We have a way of getting content to the community through the blog, subscriber tools, SCADApedia, … We also have minimal restrictions in releasing this type of information. Many asset owners and vendor security resources find these standards efforts as one of the few public areas they can contribute in.

I regrettably conclude that our pro bono time is better spend developing that content and tools than slogging through the standards process. This is not meant to devalue those documents; they are needed and we will continue to track them. Unfortunately, the pace and results per hour spent are much lower than other available options.

I mentioned this back in March — Another hacker conference SCADA presentation. The presentation is now available for download. A quick review doesn’t show anything too groundbreaking but it was interesting to learn about an Italian project called CrISTAL (Critical Infrastructures Security Testing and Analysis Lab). From the website:

CrISTAL aims to develop security methodologies for the SCADA environments, produce active research and tools, as well as sharing information and the gained know-how, making an higher level of security into these particular IT infrastructures.

I look forward to hearing more from them in the future regarding their research and tool development. Perhaps they’ll consider submitting a paper for S4 2009.

To give our readers a taste of what Daniel and I do most days I thought I would post a little code snippet and ask you all to find the overflow (if there is one). Any discussion on the feasibility of exploiting the overflow (again if there is one) is also appreciated.

I’ll keep this one fairly simple (c/c++ pseudo code):

void foo(char *buf)
{
char *myArray;

}


• Great blog entry from the guys at Matasano on hacking a ‘toaster’ running a VxWorks OS.
• The PCSF Annual Meeting will be held on August 26 - 28 in San Diego. The call for papers/solutions is out, and an agenda and registration is forthcoming. This is our top recommendation if you can only attend one control system security event each year to get updated on the latest activities in industry, solutions, standards, regulations, …

Joshua Corman of IBM/ISS gave a presentation at Interop Las Vegas yesterday titled “Unsafe at any speed: 7 Dirty Secrets of the Security Industry”. Here’s the Network World report. The title alone is interesting – making a reference to automobile safety – especially considering some recent discussion about the relationship of security to reliability and safety. I thought it might be fun to see how these seven dirty secrets apply (or not) to the control system world. So here goes…

1. Antivirus certifications are misleading
It used to be hard to even find antivirus software on control networks but I think we’re finally getting around that curve now. The important thing to remember here is that antivirus software is not going to stop most attacks but it can keep a lot of bad things from happening. Nobody wants some latent SQL Slammer to replicate from the corporate LAN into the control network via the DMZ, for example.

2. There is no perimeter
Corman’s point here was more about lost data via laptops, thumb drives, etc… I suppose you could say something about engineering laptops potentially going in and out of the SCADA network but I think the eroding perimeter of the control network surrounds two other bigger issues. The first is the geographical dispersion in most of these systems that creates physical security and general perimeter issues. The second is abuse of the DMZ concept. When used correctly, the DMZ is an important security feature used to prevent direct communication between untrusted networks. When misused or overused, however, it can be a gaping hole.

3. Risk analysis threatens vendors
Asset owners should continually assess risk to understand where their attention should be focused as well as be aware that there is no silver bullet for good security. Hopefully this will be enough to keep them from buying products that don’t make sense for their environment. Security vendors, on the other hand, must understand the goal and function of control system environments in general, and each customer specifically, to be able to offer them something of value.

4. There is more to risk than just weak software
Excellent point here. Poor coding practices are a problem in this space, but only a part of it. The bigger problem is an “open by default” mentality that leads to bad configuration practices.

5. Compliance threatens security
There are some who feel that NERC CIP and other compliance efforts actually do more harm than good. My personal experience, however, does not corroborate this. Does compliance fix everything? No. Does it bring some attention and funding where it is needed? I believe it can.

6. Vendor blind spots allowed the Storm worm outbreak to happen
To me, this is just an extension of dirty secret number one. The social engineering aspect is certainly a valid concern. The benefit of other security technologies mentioned, particularly anomaly detection, is something that has some potential in control networks.

7. Security has grown well past do-it-yourself
The Network World article quotes Corman and says this:

Security vendors try to convince businesses that security is so complex that they cannot possibly do it alone, Corman says. But the security needs of businesses are so individual that merely choosing a product is not enough. “It’s not enough to have the right tool. It needs to be installed and configured properly for the environment,” he says…

This may apply as much to the IT-Operations relationship as it does to Vendor-Customer. In some organizations, IT is the pushing for better security, dragging the operations and control systems personnel with them. In others, it is quite the opposite. In either case, Corman offers some pertinent advice.

I’m always interested to see what can be extended and applied to our community. While I don’t agree with all of Corman’s dirty secrets, he does offer some interesting points that, if nothing else, provide good food for thought.