Comment from Jake Brodsky
Time: April 22, 2007, 6:44 pm

I agree Dale. The threat from a Red Team is real. In an actual Orange or Red alert we would disconnect our SCADA system from everything. If someone exposes a control system security flaw, I’ll make my best effort to patch it in a timely fashion. But I’m not capable of hardening our system to the point where I can anticipate and deal with such attacks.

In the scheme of every day threats, this one doesn’t rate. We’re paid to defend against the likely and knowable threats –not the possibility of attack by a nation state. I can’t harden my control system against another nation state. I can’t put our operators through that degree of security. I have to worry about the lower tech approaches first. Who can get at the UPS for example…

We don’t have guards with submachine guns and flak vests guarding our plants. We don’t have miliary grade security software either. That’s what we pay taxes for. I hate to say this Dale, I have other far more likely scenarios to worry about.

Comment from Dale Peterson
Time: April 22, 2007, 8:22 pm

Jake - I agree this is not something asset owners can address. Rather it is something that vendors need to address based on pressure from asset owners and regulatory agencies.

It would be interesting to ask your vendors what their software development process is and compare the security measures to the well documented methodology by Microsoft or others. Let’s look at the security requirements in the requirements document. Can I see a test plan and results? Does the test plan cover positive and negative testing? How about testing third party components?

We have been very disappointed in the answers to these questions. I’m sure some vendors do a better job than others, but it is one of the reasons we are helping, full disclosure on a paid basis, to push the Achilles Controller certification.

Comment from Jake Brodsky
Time: April 22, 2007, 10:53 pm

I think it’s good to check these systems for flaws, Dale. I don’t believe I’ve ever said otherwise. The sticky issue is figuring out what one should do with this information, particularly if the system is not new, and has new releases available.

For example, I can run any major release of Rockwell PLC firmware from 10 through 15 on a controller. Should I always run the latest and greatest? I say no. If we have safety issues that we’ve tested for and found an earlier release to be functional, we may not want to go through that testing again just to bump up the release number –unless there is a very valid reason for doing so.

I have hundreds of PLCs out there in the field. I can’t update them all remotely, even if I wanted to. Despite bench testing everything about a new release, I still need an operator on site to validate the behavior of the new code. This gets very expensive. Isolation is often easier.

I guess what I’m saying is that while such testing at the manufacturing and safety certification levels is useful and productive, an asset owner would have a much more difficult time justifying it.

This goes back to my assertion that security has to be designed in to a system from the start. We can’t simply bolt it on after the fact. This fact of life is very hard for many people who come from IT to grasp. Software and even firmware is supposed to be something you can update. Yet, we have concerns that are alien to most outside the plant floor. We have tested an earlier release and found it adequate. Are we going to make good the victim of better? Have any of these folks ever understood the real reason why managers often talk of “shooting the engineer” to get the job done?

So on a larger note: I think this talk of convergence of Control systems and IT needs to be examined more critically. Many who think they need real time data, probably don’t. We all think it’s cool to have up to the second data on your desk. But at what cost? And where does the improved productivity or legal compliance come from by having this data?

This is where security needs to start from. We need to discuss vertical applications, information flows, and then discuss what appliances we need to effect these requirements. Then, I think the OEMs should make regular testing of each new release of firmware or software. Asset Owners need to be advised of the results of this test.

Chances are that we won’t care in most cases to make an update. However, there may be some legitimate reasons why we’d want to carefully evaluate new firmware or software and then carefully propagate those updates to the field.

Comment from Ron Southworth
Time: April 22, 2007, 11:09 pm

As you say, all you can do Jake is have your emergency prepairedness processes and proceedures up to date and solid, hopefully have established some good lines of communication between your interdependant services and support organisations that hopefully include the government, police etc.

Dale you raise a valid point on quality control - let’s face it most of the security flaw disclosures relate to quality control or testing “oversights”. I asked a vendor here about three years ago re their quality control and to see if they had removed their back door to the product.

Go away and No were the answers! Needless to say I suspect this vendor is going to go the way of the dodo!

Security requirements - I am really waiting for the next update on The Common Procurement Language.

Looking at the to do list the next one should have sufficient coverage to really make people look at utilising the language in a serious way. What I am finding is that some vendors are looking at teh document as it presently stands and actually planning on addressing this in a meaningful way.

I guess I am saying Dale is that there are some that are ready and willing to participate and some that are not. So far the ones that are not it is thru a lack of vision and not related to their market share. Any tools that can improve the quality of the product are a good thing.

Comment from Ralph Langner
Time: April 23, 2007, 4:36 am

Dale makes a good point about our threat landscape. I agree with him that it would be quite easy to find someone who implements exploits for various widely used control systems. Some $ figure in the lower five digits region will do.

I disagree with Jake that asset owners have other things to worry about because this scenario would be unlikely. To me, it appears unlikely as a targeted attack against a specific facility. It appears likely, however, that this scenario will materialize in the malware category, where a group of evildoers will try to knock out thousands of facilities with a small piece of code, purchased for little money. I believe that the SCADA malware threat is underestimated.

Comment from Ron Southworth
Time: April 23, 2007, 7:49 am

Hi Ralph.

Maybe you guys have other data?

I harken back to the statistics from the BCIT database that point to external forms of attack vectors being 20% and internal being 80% (I think these figures are reasonably current ?

If a utility invested in addressing the 80% and did absolutely nothing about the 20% they would certainly have to be a lot better off in reducing their risk exposure and it would probably be a very good return for investment. I am not by any means dismissing you or Dale that the external threat is not there and that the motivator is not that hard to acheive.

Where is the data to support the likelyhood of this type of attack being more important then the internal ones to deal with.

The principals of good defence in depth is not just good for security but it also makes the system more robust and available if the message is adhered to.

There would be a greater likelihood of a Windows targeted attack having greater market impact for certain as an external threat source there are just more boxes sitting on control LAN’s using MS windows.

I don’t think that malware is under estimated as a threat vector there is just not enough data to support spending the effort and dollars when there are very cost effective methods to mitigate malware and the 80% is the more common day to day problem!

Comment from Rob Lewis
Time: April 23, 2007, 12:45 pm

@Jake

” We don’t have military grade security software either.”

Although this whole discussion has value, much of this has been said many times before.

Our approach has been the development of a security sub-system that addresses the problems with previous high and medium security systems, including multi-level security environments, that have made them a non-starter for commercial use.

We have developed new security sub-systems that can be integrated with low security systems (read windows) to elevate them to medium to high security status. The technology is cost effective and manageable.

This level of security is not confined to a single host machine, lending itself to entire networks.

This approach offers protection to existing SCADA environments now, and protects systems and data from vulnerabilties, regardless of patch-state.

I do not wish to cross the line and appear to be selling on this blog. However, if there was interest, I would post the link to a theoretical white paper if someone requested it, and only if Mr. Peterson approved, for the purpose of advancing other options for discussion.

Comment from Terry Martin
Time: April 23, 2007, 3:12 pm

This is a good example of the “Cost to Break” curve intersecting with the realization of an incident.

ref:
http://www.giac.org/resources/whitepaper/management/332.php

Comment from Jake Brodsky
Time: April 23, 2007, 10:27 pm

Terry and Rob, I appreciate the time you’re taking to reply on this. I’ve designed my network around the notion of defense in depth. I don’t have as many places for firewall appliances as you guys probably would like to see, but I do have them.

Here are the things we need to consider: Firewalls do have a propagation time. Are they deterministic? If I send a DNP over UDP packet for time sync through one of these things, will it seriously affect my synchronization time? How many of these things can I get through before it does become a factor?

Keep in mind, some applications really need to maintain synchronization within a few milliseconds. So this isn’t idle speculation.

Second, the Cost to Break curve is something we’ve seen before. We think of it in terms of the old joke about not having to outrun the bear; you only need to outrun your fellow campers. And in fact, physical security is often lagging as well. In the middle of all this we still need to run a real time industrial process.

All security systems have overheads. And at some point the overhead is greater than the cost of a typical attack. Somewhere between the Cost to Break and the cost of attack is a suitable answer to the security question.

Finally, I know you guys cringe every time I talk about patching policies. Let me be the first to explain that we try very hard in our newer systems to allow for patching. However that gets expensive too, because our process programs need to be more distributed and we need far more error checking.

Right now the software cost isn’t too high. If the thing falls on the floor, it’s not exactly a “so what?” event, but it is easily fixed. The more complex a security system gets, the more damaging an attack can be because restoral isn’t as simple as it was before.

At some point, we’re going to scream “Uncle!” at the excessive complexity of what is being created around us. Unlike many applications, we must enlist the operators to help us. We must keep these systems nimble so that we can easily modify the code too. The more security gets in the way of dynamic updates to what was simple code before, the less desirable the control system will be to modify.

And at that point the economics of this whole edifice goes down the drain. We do the simple and cost effective measures. We do what we can to design in places to implement unobtrusive security. However, there are limits. The system has to remain usable, or the operators will ignore it and bypass it. And then, no matter how secure you think you are, it won’t matter.

Comment from Rob Lewis
Time: April 24, 2007, 12:00 pm

Thank you for your thorough response Jake. Of course the current model of security is adding complexity and more overhead to the IT environment. It is based on a number of inherent flawed assumptions about security design that ultimately limits the degree of success that will be achievable.

What I am talking about is core security, which is really the missing essential layer in a layered defense, which has been missing from the start really, and a proactive model of security rather than reactive.

I don’t want you to think that we are the typical IT outfit that is totally unaware of the special needs of this vertical. This technology has a predictable latency, (was designed for embedded RTOS systems as well).

Although I think it has the potential to make all future control systems trusted rather than merely hardened, it has immediate applicability in enforcing a pre-emptive lockdown of the low grade security intermediary commerical systems that sit between control systems and the internet.

The best thing about it though, is that it does not impede business operations, and does not require a third degree black belt in IT security to run it.

I would even go out on a limb and say that it in many ways would even pay for itself over time, but that thinking is from traditional IT deployments so I will have to temper that thought until we make more headway into this area.

Comment from Terry Martin
Time: April 24, 2007, 6:32 pm

The way to deal with complexity is reduce it. There is not much point trying to secure high level processes, if failure points exist in low level processes.

The problem has been introduced with the IP stack, so the stack itself needs to be reliable or nothing up-level will be reliable.

Establishing a standard automated test that can grind through every conceivable combination of variables encountered in a production network to certify conformance with basic OSI protocol requirements will set the foundation for up-level considerations.

I, for one, would never accept a supply side certification. The fact a device has been released is the only supply side statement of conformity that matters.

What I want is to know that in my production environment the thing is always going to do what it’s supposed to, and only what its supposed to, and I want it verified independently before I waste any time considering it.

That means it’s up to the developer to get it done independently.